Intraoperative display for surgical systems

ABSTRACT

An augmented reality display system used during a surgical procedure is disclosed. The augmented reality display system includes an imaging device to capture a real image of a surgical area during the surgical procedure. An augmented reality display presents an overlay of an operational aspect of a surgical instrument being actively visualized onto the real image of the surgical area. The overlay combines aspects of tissue interaction in the surgical area with functional data received from the surgical instrument. A processor receives the functional data from the surgical instrument, determines the overlay related to the operational aspect of the surgical instrument, and combines the aspect of the tissue in the surgical area with the functional data received from the surgical instrument.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 U.S.C. § 119(e) to U.S.Provisional Patent Application No. 63/174,674, titled HEADS UP DISPLAY,filed Apr. 14, 2021 and to U.S. Provisional Patent Application No.63/284,326, titled INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS, filedNov. 30, 2021, the disclosure of each of which is herein incorporated byreference in its entirety.

BACKGROUND

This disclosure relates to apparatuses, systems, and methods forproviding an augmented reality interactive experience during a surgicalprocedure. During a surgical procedure it would be desirable to providean augmented reality interactive experience of a real-world environmentwhere objects that reside in the real world are enhanced by overlayingcomputer-generated perceptual information, sometimes across multiplesensory modalities, including visual, auditory, haptic, somatosensory,and olfactory. In the context of this disclosure, images of a surgicalfield and surgical instruments and other objects appearing in thesurgical field are enhanced by overlaying computer-generated visual,auditory, haptic, somatosensory, olfactory, or other sensory informationonto the real world images of the surgical field and instruments orother objects appearing in the surgical field. The images may bestreamed in real time or may be still images.

Real world surgical instruments include a variety of surgical devices.Energy based surgical devices include, without limitation,radio-frequency (RF) based monopolar and bipolar electrosurgicalinstruments, ultrasonic surgical instruments, combination RFelectrosurgical and ultrasonic instruments, combination RFelectrosurgical and mechanical staplers, among others. Surgical staplerdevices are surgical instruments used to cut and staple tissue in avariety of surgical procedures, including bariatric, thoracic,colorectal, gynecologic, urologic and general surgery.

SUMMARY

In various instances, this disclosure provides an augmented realitydisplay system for use during a surgical procedure. The augmentedreality display system comprises an imaging device to capture a realimage of a surgical area during the surgical procedure. An augmentedreality display presents an overlay of an operational aspect of asurgical instrument being actively visualized onto the real image of thesurgical area. The overlay combines aspects of tissue interaction in thesurgical area with functional data received from the surgicalinstrument. A processor receives the functional data from the surgicalinstrument, determines the overlay related to the operational aspect ofthe surgical instrument, and combines the aspect of the tissue in thesurgical area with the functional data received from the surgicalinstrument.

In various instances, this disclosure provides an augmented realitydisplay system for use during a surgical procedure. The augmentedreality display system comprises an imaging device to capture a realimage of a surgical area during the surgical procedure. An augmentedreality display presents an overlay of an operational aspect of asurgical instrument being actively visualized onto the real image of thesurgical area. The overlay combines aspects of tissue interaction in thesurgical area with functional data from the surgical instrument. Aprocessor receives the functional data from the surgical instrument,determines the overlay related to the operational aspect of the surgicalinstrument, combines the aspect of the tissue in the surgical area withthe functional data received from the surgical instrument, andconfigures the overlay.

In various instance, this disclosure provides an augmented realitydisplay system for use during a surgical procedure. The augmentedreality display system comprises an imaging device to capture a realimage of a surgical area during the surgical procedure. An augmentedreality display presents an overlay of an operational aspect of asurgical instrument being actively visualized onto the real image of thesurgical area. The overlay combines aspects of tissue interaction in thesurgical area with functional data from the surgical instrument. Aprocessor receives the functional data from the surgical instrument,determines the overlay related to the operational aspect of the surgicalinstrument, combines the aspect of the tissue in the surgical area withthe functional data received from the surgical instrument, and displaysdynamic status of the surgical instrument.

FIGURES

The various aspects described herein, both as to organization andmethods of operation, together with further objects and advantagesthereof, may best be understood by reference to the followingdescription, taken in conjunction with the accompanying drawings asfollows.

FIG. 1 is a block diagram of a computer-implemented interactive surgicalsystem, according to one aspect of this disclosure.

FIG. 2 is a surgical system being used to perform a surgical procedurein an operating room, according to one aspect of this disclosure.

FIG. 3 is a surgical hub paired with a visualization system, a roboticsystem, and an intelligent instrument, according to one aspect of thisdisclosure.

FIG. 4 illustrates a surgical data network comprising a modularcommunication hub configured to connect modular devices located in oneor more operating theaters of a healthcare facility, or any room in ahealthcare facility specially equipped for surgical operations, to thecloud, according to one aspect of this disclosure.

FIG. 5 illustrates a computer-implemented interactive surgical system,according to one aspect of this disclosure.

FIG. 6 illustrates a surgical hub comprising a plurality of modulescoupled to the modular control tower, according to one aspect of thisdisclosure.

FIG. 7 illustrates an augmented reality (AR) system comprising anintermediate signal combiner positioned in the communication pathbetween an imaging module and a surgical hub display, according to oneaspect of this disclosure.

FIG. 8 illustrates an augmented reality (AR) system comprising anintermediate signal combiner positioned in the communication pathbetween an imaging module and a surgical hub display, according to oneaspect of this disclosure.

FIG. 9 illustrates an augmented reality (AR) device worn by a surgeon tocommunicate data to the surgical hub, according to one aspect of thisdisclosure.

FIG. 10 illustrates a system for augmenting surgical instrumentinformation using an augmented reality display, according to one aspectof this disclosure.

FIG. 11 is an augmented image of a live feed of a surgical areavisualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating appropriate tissue captured between jawsof a surgical instrument end effector as a tissue aspect, according toone aspect of this disclosure.

FIG. 12 is an augmented image of a live feed of a surgical areavisualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating tissue position proximally capturedbetween of a surgical instrument end effector as a tissue aspect,according to one aspect of this disclosure.

FIG. 13 is an augmented image of a live feed of a surgical area asvisualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating tissue insufficiently captured betweenjaws of a surgical instrument end effector as a tissue aspect, accordingto one aspect of this disclosure.

FIG. 14 is another augmented image of a live feed of a surgical area asvisualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating tissue insufficiently captured betweenjaws of a surgical instrument end effector as a tissue aspect, accordingto one aspect of this disclosure.

FIG. 15 is another augmented image of a live feed of a surgical area asvisualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating tissue insufficiently captured betweenjaws of a surgical instrument end effector as a tissue aspect, accordingto one aspect of this disclosure.

FIG. 16 is an augmented image of a live feed of a surgical area asvisualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating tissue tension as a tissue aspect,according to at least one aspect of this disclosure.

FIG. 17 is another augmented image of a live feed of a surgical area asvisualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating tissue tension as a tissue aspect,according to one aspect of this disclosure.

FIG. 18 is a plurality of graphic images indicating jaw closure positionas an operational aspect of a surgical instrument as shown in FIGS.11-17, according to one aspect of this disclosure.

FIG. 19 is an augmented image of a live feed of a surgical area asvisualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating jaw closure position as an operationalaspect of a surgical instrument, according to one aspect of thisdisclosure.

FIG. 20 is an augmented image of a live feed of the surgical area shownin FIG. 19 showing a fully closed surgical instrument end effector and agraphical alert overlay showing jaw closed position superimposed on theend effector, according to one aspect of this disclosure.

FIG. 21 is an augmented image of a live feed of a surgical area asvisualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating clamping on metal or foreign object as anoperational aspect of a surgical instrument, according to one aspect ofthis disclosure.

FIG. 22 is an augmented image of a live feed of a surgical area asvisualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating device residual heat warning whereoverheating is an operational aspect of a surgical instrument, accordingto one aspect of this disclosure.

FIG. 23 is an augmented image of a live feed of a surgical area asvisualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating tissue movement and flow is a tissueaspect, according to one aspect of this disclosure.

FIG. 24 is an augmented image of a live feed of a surgical area asvisualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating the geometric relationship of thetransections to tissue and other firings is a tissue aspect, accordingto one aspect of this disclosure.

FIG. 25 is an image showing anvil orientation communication as anoperational aspect of a surgical instrument, according to one aspect ofthis disclosure.

FIG. 26 is an image showing detected tissue thickness in the jaws of asurgical instrument end effector where tissue thickness is a tissueaspect, according to one aspect of this disclosure.

FIG. 27 is an image of temperature gradient display graphics for anultrasonic instrument, according to one aspect of this disclosure.

FIG. 28 is an image of temperature icon display graphics for anultrasonic instrument, according to one aspect of this disclosure.

FIG. 29 is an image of an ultrasonic blade temperature graphic elementsmapped to an end effector jaw position, according to one aspect of thisdisclosure.

FIG. 30 is an image of an ultrasonic generator power level displaygraphic, according to one aspect of this disclosure.

FIG. 31 is an image of an ultrasonic generator power level displaygraphic with a pop-up warning graphic indicating that the ultrasonic endeffector jaw is overstuffed, according to at aspect of this disclosure.

FIG. 32 is an image of an ultrasonic generator power level displaygraphic with a pop-up warning graphic indicating ultrasonic end effectorjaw heat, according to one aspect of this disclosure.

FIG. 33 is an image of an electrosurgical generator display graphic witha pop-up warning graphic indicating electrosurgical seal qualityprediction, according to one aspect of this disclosure.

FIG. 34 is an image of a surgical stapler reload feedback, according toone aspect of this disclosure.

FIG. 35 is an image of a surgical stapler precompression countdown,according to one aspect of this disclosure.

FIG. 36 is a system diagram of a surgical suite comprising a surgicalmonitor with intraoperative data display of a surgical area, accordingto one aspect of this disclosure.

FIG. 37 is an augmented image of a live feed of a surgical area asvisualized through a laparoscopic camera during a minimally invasivesurgical procedure displayed on an intraoperative data display,according to one aspect of this disclosure.

FIG. 38 is a detailed view of the case information panel overlay shownin FIG. 37, according to one aspect of this disclosure.

FIG. 39 is a detailed view of the systems notifications panel overlayshown in FIG. 37, according to one aspect of this disclosure.

FIG. 40 is an image of several examples of systems notifications paneloverlays, according to one aspect of this disclosure.

FIG. 41 is a detailed view of the device panels overlay shown in FIG.37, according to one aspect of this disclosure.

FIG. 42 is an augmented image of a live feed of a surgical area asvisualized through a laparoscopic camera during a minimally invasivesurgical procedure displayed on an intraoperative data display,according to one aspect of this disclosure.

FIG. 43 is a schematic view of an energy device image panelarchitecture, according to one aspect of this disclosure.

FIG. 44 is an image of supplemental device alerts/warning/information ina stacked configuration, according to one aspect of this disclosure.

FIG. 45 is an image of supplemental device alerts/warning/information inan expanded configuration, according to one aspect of this disclosure.

FIG. 46 is an instrument state image panel showing how instrument panelstates change dynamically to show state changes such as deviceactivation or power level adjustment, according to one aspect of thisdisclosure.

FIG. 47 is a system diagram of translating generator alerts and warningsto a laparoscopic monitor and displayed on a local interface, accordingto one aspect of this disclosure.

FIG. 48 is a diagram of a series of screens of existing alerts shown ona current generator that are transmitted to a surgical hub, which thendisplays them as a series of screens on a local interface, according toone aspect of this disclosure.

FIG. 49 is a schematic diagram of a system comprising a generator incommunication with a digital hub, which then displays screen data andalert data on a local interface such as a laparoscopic screen, accordingto one aspect of this disclosure.

FIG. 50 is an augmented image of a live feed of a surgical area asvisualized through a laparoscopic camera during a minimally invasivesurgical procedure displayed on an intraoperative data display,according to one aspect of this disclosure.

FIG. 51 is a display screen showing an intraoperative data displaycomprising a secondary bottom edge configurable panel, according to oneaspect of this disclosure.

FIG. 52 is an alternative bottom edge configurable panel, according toone aspect of this disclosure.

FIG. 53 is a display screen showing an intraoperative data displaycomprising a secondary top left corner configurable panel, according toone aspect of this disclosure.

FIG. 54 is a display screen showing an intraoperative data displaycomprising a secondary top center configurable panel, according to oneaspect of this disclosure.

FIG. 55 is display screen showing an intraoperative data displaycomprising a secondary side edge configurable panel, according to oneaspect of this disclosure.

FIG. 56 is a series of image panels displaying device troubleshootinginformation, according to one aspect of this disclosure.

FIG. 57 is a series of image panels displaying articulating surgicalstapler features, according to one aspect of this disclosure.

FIG. 58 is an alert/warning/information image panel displaying that thearticulation limit has been reached, according to one aspect of thisdisclosure.

FIG. 59 is an alert/warning/information image panel displaying that thedevice is in lockout mode, according to one aspect of this disclosure.

FIG. 60 is an alert/warning/information image panel displaying that thedevice cannot articulate when jaws are closed, according to one aspectof this disclosure.

FIG. 61 is a device image panel showing articulating surgical staplerfeatures, according to one aspect of this disclosure.

FIG. 62 is a stacked alert/warning/information image panel displayed ina stacked configuration with device alert displaying that thearticulation limit has been reached, according to one aspect of thisdisclosure.

FIG. 63 is a schematic diagram of a system comprising a surgical staplerin communication with a digital hub over Bluetooth to execute analgorithm for the countdown timer image panel shown in FIGS. 57 and 61,according to one aspect of this disclosure.

FIG. 64 is a series of device image panels/alerts displaying ultrasonicinstrument features, according to one aspect of this disclosure.

FIG. 65 is a chart describing pairing a surgical stapler instrument,according to one aspect of this disclosure.

FIG. 66 is an image of a screen displaying pairing devices information,according to one aspect of this disclosure.

FIG. 67 is an image of a wireless surgical device comprising a uniqueidentifier for pairing wireless devices, according to one aspect of thisdisclosure.

FIG. 68 is an image of a screen displaying a link to optimal deviceperformance (ODP) guide images or other electronic instructions for use(e-IFU), according to one aspect of this disclosure.

FIG. 69 is a diagram of an augmented reality method employing a surgicalinstrument and an augmented reality display for use during a surgicalprocedure, according to one aspect of this disclosure.

FIG. 70 is a diagram of an augmented reality method employing a surgicalinstrument and an augmented reality display for use during a surgicalprocedure, according to one aspect of this disclosure.

FIG. 71 is an image of a staff view screen displaying customizedoverlays information, according to one aspect of this disclosure.

FIG. 72 is an image of a staff view screen displaying detailedcustomization pop-up information, according to one aspect of thisdisclosure.

FIG. 73 is an image of a staff view screen displaying staff viewtroubleshooting pop-up information, according to one aspect of thisdisclosure.

FIG. 74 is an image of a primary surgical display interactions screendisplaying primary surgical display interactions, according to oneaspect of this disclosure.

FIG. 75 illustrates a timeline of a situational awareness surgicalprocedure, according to one aspect of this disclosure.

Corresponding reference characters indicate corresponding partsthroughout the several views. The exemplifications set out hereinillustrate various disclosed embodiments, in one form, and suchexemplifications are not to be construed as limiting the scope thereofin any manner.

DESCRIPTION

Applicant of the present application owns the following U.S. PatentApplications filed concurrently herewith, the disclosures of each ofwhich is herein incorporated by reference in its entirety:

-   -   U.S. Patent Application, titled METHOD FOR INTRAOPERATIVE        DISPLAY FOR SURGICAL SYSTEMS; Attorney Docket No.        END9352USNP1/210120-1M;    -   U.S. Patent Application, titled Utilization of surgical data        values and situational awareness to control the overlay in        surgical field view; Attorney Docket No. END9352USNP2/210120-2;    -   U.S. Patent Application, titled SELECTIVE AND ADJUSTABLE MIXED        REALITY OVERLAY IN SURGICAL FIELD VIEW; Attorney Docket No.        END9352USNP3/210120-3;    -   U.S. Patent Application, titled RISK BASED PRIORITIZATION OF        DISPLAY ASPECTS IN SURGICAL FIELD VIEW; Attorney Docket No.        END9352USNP4/210120-4;    -   U.S. Patent Application, titled SYSTEMS AND METHODS FOR        CONTROLLING SURGICAL DATA OVERLAY; Attorney Docket No.        END9352USNP5/210120-5;    -   U.S. Patent Application, titled SYSTEMS AND METHODS FOR CHANGING        DISPLAY OVERLAY OF SURGICAL FIELD VIEW BASED ON TRIGGERING        EVENTS; Attorney Docket No. END9352USNP6/210120-6;    -   U.S. Patent Application, titled CUSTOMIZATION OF OVERLAID DATA        AND CONFIGURATION; Attorney Docket No. END9352USNP7/210120-7;    -   U.S. Patent Application, titled INDICATION OF THE COUPLE PAIR OF        REMOTE CONTROLS WITH REMOTE DEVICES FUNCTIONS; Attorney Docket        No. END9352USNP8/210120-8;    -   U.S. Patent Application, titled COOPERATIVE OVERLAYS OF        INTERACTING INSTRUMENTS WHICH RESULT IN BOTH OVERLAYS BEING        EFFECTED; Attorney Docket No. END9352USNP9/210120-9;    -   U.S. Patent Application, titled ANTICIPATION OF INTERACTIVE        UTILIZATION OF COMMON DATA OVERLAYS BY DIFFERENT USERS; Attorney        Docket No. END9352USNP10/210120-10;    -   U.S. Patent Application, titled MIXING DIRECTLY VISUALIZED WITH        RENDERED ELEMENTS TO DISPLAY BLENDED ELEMENTS AND ACTIONS        HAPPENING ON-SCREEN AND OFF-SCREEN; Attorney Docket No.        END9352USNP11/210120-11;    -   U.S. Patent Application, titled SYSTEM AND METHOD FOR TRACKING A        PORTION OF THE USER AS A PROXY FOR NON-MONITORED INSTRUMENT;        Attorney Docket No. END9352USNP12/210120-12;    -   U.S. Patent Application, titled UTILIZING CONTEXTUAL PARAMETERS        OF ONE OR MORE SURGICAL DEVICES TO PREDICT A FREQUENCY INTERVAL        FOR DISPLAYING SURGICAL INFORMATION; Attorney Docket No.        END9352USNP13/210120-13;    -   U.S. Patent Application, titled COOPERATION AMONG MULTIPLE        DISPLAY SYSTEMS TO PROVIDE A HEALTHCARE USER CUSTOMIZED        INFORMATION; Attorney Docket No. END9352USNP14/210120-14;    -   U.S. Patent Application, titled ADAPTATION AND ADJUSTABILITY OR        OVERLAID INSTRUMENT INFORMATION FOR SURGICAL SYSTEMS; Attorney        Docket No. END9352USNP16/210120-16; and    -   U.S. Patent Application, titled MIXED REALITY FEEDBACK SYSTEMS        THAT COOPERATE TO INCREASE EFFICIENT PERCEPTION OF COMPLEX DATA        FEEDS; Attorney Docket No. END9352USNP17/210120-17.

Applicant of this application owns the following U.S. PatentApplications, the disclosure of each of which is herein incorporated byreference in its entirety:

-   -   U.S. patent application Ser. No. 16/209,423, titled METHOD OF        COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY        DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS, now U.S.        Patent Publication No. US-2019-0200981-A1;    -   U.S. patent application Ser. No. 16/209,453, titled METHOD FOR        CONTROLLING SMART ENERGY DEVICES, now U.S. Patent Publication        No. US-2019-0201046-A1.

Before explaining various aspects of surgical devices and generators indetail, it should be noted that the illustrative examples are notlimited in application or use to the details of construction andarrangement of parts illustrated in the accompanying drawings anddescription. The illustrative examples may be implemented orincorporated in other aspects, variations and modifications, and may bepracticed or carried out in various ways. Further, unless otherwiseindicated, the terms and expressions employed herein have been chosenfor the purpose of describing the illustrative examples for theconvenience of the reader and are not for the purpose of limitationthereof. Also, it will be appreciated that one or more of thefollowing-described aspects, expressions of aspects, and/or examples,can be combined with any one or more of the other following-describedaspects, expressions of aspects and/or examples.

Various aspects are directed to onscreen displays for surgical systemsfor a variety of energy and surgical stapler based medical devices.Energy based medical devices include, without limitation,radio-frequency (RF) based monopolar and bipolar electrosurgicalinstruments, ultrasonic surgical instruments, combination RFelectrosurgical and ultrasonic instruments, combination RFelectrosurgical and mechanical staplers, among others. Surgical staplerdevices include and combined surgical staplers with electrosurgicaland/or ultrasonic devices. Aspects of the ultrasonic surgical devicescan be configured for transecting and/or coagulating tissue duringsurgical procedures, for example. Aspects of the electrosurgical devicescan be configured for transecting, coagulating, sealing, welding and/ordesiccating tissue during surgical procedures, for example. Aspects ofthe surgical stapler devices can be configured for transecting andstapling tissue during surgical procedures and in some aspects, thesurgical stapler devices may be configured to delivery RF energy to thetissue during surgical procedures. Electrosurgical devices areconfigured to deliver therapeutic and/or nontherapeutic RF energy to thetissue. Elements of surgical staplers, electrosurgical, and ultrasonicdevices may be used in combination in a single surgical instrument.

In various aspects, the present disclosure provides onscreen displays ofreal time information to the OR team during a surgical procedure. Inaccordance with various aspects of the present disclosure, many new andunique onscreen displays are provided to display onscreen a variety ofvisual information feedback to the OR team. According to the presentdisclosure, visual information may comprise one or more than one ofvarious visual media with or without sound. Generally, visualinformation comprises still photography, motion picture photography,video or audio recording, graphic arts, visual aids, models, display,visual presentation services, and the support processes. The visualinformation can be communicated on any number of display options such asthe primary OR screen, the energy or surgical stapler device itself, atablet, augmented reality glasses, among others, for example.

In various aspects, the present disclosure provides a large list ofpotential options to communicate visual information in real time to theOR team, without overwhelming the OR team with too much visualinformation. For example, in various aspects, the present disclosureprovides onscreen displays of visual information to enable the surgeon,or other members of the OR team, to selectively activate onscreendisplays such as icons surrounding the screen option to manage a wealthof visual information. One or a combination of factors can be used todetermine the active display, these may include energy based (e.g.,electrosurgical, ultrasonic) or mechanical based (e.g., staplers)surgical devices in use, the estimated risk associated with a givendisplay, the experience level of the surgeon and the surgeons' choiceamong other things. In other aspect, the visual information maycomprises rich data overlaid or superimposed into the surgical field ofview to manage the visual information. In various aspects describedhereinbelow, comprise superimposed imagery that requires video analysisand tracking to properly overlay the data. Visual information datacommunicated in this manner, as opposed to static icons, may provideadditional useful visual information in a more concise and easy tounderstand way to the OR team.

In various aspects, the present disclosure provides techniques forselectively activating onscreen displays such as icons surrounding thescreen to manage visual information during a surgical procedure. Inother aspects, the present disclosure provides techniques fordetermining the active display using one or a combination of factors. Invarious aspects, the techniques according to the resent disclosure maycomprise selecting the energy based or mechanical based surgical devicein use as the active display, estimating risk associated with a givendisplay, utilizing the experience level of the surgeon or OR team makingthe selection, among other things.

In other aspects, the techniques according to the present disclosure maycomprise overlaying or superimposing rich data onto the surgical fieldof view to manage the visual information. A number of the displayarrangements described by the present disclosure involve overlayingvarious visual representations of surgical data onto a livestream of asurgical field. As used herein the term overlay comprises a translucentoverlay, a partial overlay, and/or a moving overlay. Graphical overlaysmay be in the form of a transparent graphic, semitransparent graphic, oropaque graphic, or a combination of transparent, semitransparent, andopaque elements or effects. Moreover, the overlay can be positioned on,or at least partially on, or near an object in the surgical field suchas, for example, an end effector and/or a critical surgical structure.Certain display arrangements may comprise a change in one or moredisplay elements of an overlay including a change in color, size, shape,display time, display location, display frequency, highlighting, or acombination thereof, based on changes in display priority values. Thegraphical overlays are rendered on top of the active display monitor toconvey important information quickly and efficiently to the OR team.

In other aspects, the techniques according to the present disclosure maycomprise superimposing imagery that requires analyzing video andtracking for properly overlaying the visual information data. In otheraspects, the techniques according to the present disclosure may comprisecommunicating rich visual information, as opposed to simple staticicons, to provide additional visual information to the OR team in a moreconcise and easy to understand manner. In other aspects, the visualoverlays may be used in combination with audible and/or somatosensoryoverlays such as thermal, chemical, and mechanical devices, andcombinations thereof.

The following description is directed generally to apparatuses, systems,and methods that provide an augmented reality (AR) interactiveexperience during a surgical procedure. In this context, images of asurgical field and surgical instruments and other objects appearing inthe surgical field are enhanced by overlaying computer-generated visual,auditory, haptic, somatosensory, olfactory, or other sensory informationonto the real world images of the surgical field, instruments, and/orother objects appearing in the surgical field. The images may bestreamed in real time or may be still images. Augmented reality is atechnology for rendering and displaying virtual or “augmented” virtualobjects, data, or visual effects overlaid on a real environment. Thereal environment may include a surgical field. The virtual objectsoverlaid on the real environment may be represented as anchored or in aset position relative to one or more aspects of the real environment. Ina non-limiting example, if a real world object exits the realenvironment field of view, a virtual object anchored to the real worldobject would also exit the augmented reality field of view.

A number of the display arrangements described by the present disclosureinvolve overlaying various visual representations of surgical data ontoa livestream of a surgical field. As used herein the term overlayingcomprises a translucent overlay, a partial overlay, and/or a movingoverlay. Moreover, the overlay can be positioned on, or at leastpartially on, or near an object in the surgical field such as, forexample, an end effector and/or a critical surgical structure. Certaindisplay arrangements may comprise a change in one or more displayelements of an overlay including a change in color, size, shape, displaytime, display location, display frequency, highlighting, or acombination thereof, based on changes in display priority values.

As described herein AR is an enhanced version of the real physical worldthat is achieved through the use of digital visual elements, sound, orother sensory stimuli delivered via technology. Virtual Reality (VR) isa computer-generated environment with scenes and objects that appear tobe real, making the user feel they are immersed in their surroundings.This environment is perceived through a device known as a VirtualReality headset or helmet. Mixed reality (MR) and AR are both consideredimmersive technologies, but they aren't the same. MR is an extension ofMixed reality that allows real and virtual elements to interact in anenvironment. While AR adds digital elements to a live view often byusing a camera, an MR experience combines elements of both AR and VR,where real-world and digital objects interact.

In an AR environment, one or more computer-generated virtual objects maybe displayed along with one or more real (i.e., so-called “real world”)elements. For example, a real-time image or video of a surroundingenvironment may be shown on a computer screen display with one or moreoverlaying virtual objects. Such virtual objects may providecomplementary information relating to the environment or generallyenhance a user's perception and engagement with the environment.Conversely, the real-time image or video of the surrounding environmentmay additionally or alternatively enhance a user's engagement with thevirtual objects shown on the display.

The apparatuses, systems, and methods in the context of this disclosureenhance images received from one or more imaging devices during asurgical procedure. The imaging devices may include a variety of scopesused during non-invasive and minimally invasive surgical procedures, anAR device, and/or a camera to provide images during open surgicalprocedures. The images may be streamed in real time or may be stillimages. The apparatuses, systems, and methods provide an augmentedreality interactive experience by enhancing images of the real worldsurgical environment by overlaying virtual objects or representations ofdata and/or real objects onto the real surgical environment. Theaugmented reality experience may be viewed on a display and/or an ARdevice that allows a user to view the overlaid virtual objects onto thereal world surgical environment. The display may be located in theoperating room or remote from the operating room. AR devices are worn onthe head of the surgeon or other operating room personnel and typicallyinclude two stereo-display lenses or screens, including one for each eyeof the user. Natural light is permitted to pass through the twotransparent or semi-transparent display lenses such that aspects of thereal environment are visible while also projecting light to make virtualobjects visible to the user of the AR device.

Two or more displays and AR devices may be used in a coordinated manner,for example with a first display or AR device controlling one or moreadditional displays or AR devices in a system with defined roles. Forexample, when activating display or an AR device, a user may select arole (e.g., surgeon, surgical assistant, nurse, etc., during a surgicalprocedure) and the display or AR device may display information relevantto that role. For example, a surgical assistant may have a virtualrepresentation of an instrument displayed that the surgeon needs toperform for a next step of a surgical procedure. A surgeon's focus onthe current step may see different information displayed than thesurgical assistant.

Although there are many known onscreen displays and alerts, thisdisclosure provides many new and unique augmented reality interactiveexperiences during a surgical procedure. Such augmented realityinteractive experiences include visual, auditory, haptic, somatosensory,olfactory, or other sensory feedback information to the surgical teaminside or outside the operating room. The virtual feedback informationoverlaid onto the real world surgical environment may be provided to anoperating room (OR) team, including personnel inside the OR including,without limitation, the operating surgeon, assistants to the surgeon, ascrub person, an anesthesiologist and a circulating nurse, among others,for example. The virtual feedback information can be communicated on anynumber of display options such as a primary OR screen display, an ARdevice, the energy or surgical stapler instrument, a tablet, augmentedreality glasses, device etc.

FIG. 1 depicts a computer-implemented interactive surgical system 1 thatincludes one or more surgical systems 2 and a cloud-based system 4. Thecloud-based system 4 may include a remote server 13 coupled to a storagedevice 5. Each surgical system 2 includes at least one surgical hub 6 incommunication with the cloud 4. For example, the surgical system 2 mayinclude a visualization system 8, a robotic system 10, and handheldintelligent surgical instruments 12, each configured to communicate withone another and/or the hub 6. In some aspects, a surgical system 2 mayinclude an M number of hubs 6, an N number of visualization systems 8,an O number of robotic systems 10, and a P number of handheldintelligent surgical instruments 12, where M, N, O, and P are integersgreater than or equal to one. The computer-implemented interactivesurgical system 1 may be configured to provide an augmented realityinteractive experience during a surgical procedure as described herein.

FIG. 2 depicts an example of a surgical system 2 to perform a surgicalprocedure on a patient lying down on an operating table 14 in a surgicaloperating room 16. A robotic system 10 is used in the surgical procedureas a part of the surgical system 2. The robotic system 10 includes asurgeon's console 18, a patient side cart 20 (surgical robot), and asurgical robotic hub 22. The patient side cart 20 can manipulate atleast one removably coupled surgical tool 17 through a minimallyinvasive incision in the body of the patient while the surgeon views thesurgical site through the surgeon's console 18 or an augmented reality(AR) device 66 worn by the surgeon. An image (e.g., still or livestreamed in real time) of the surgical site during a minimally invasiveprocedure can be obtained by a medical imaging device 24. The patientside cart 20 can manipulate the imaging device 24 to orient the imagingdevice 24. An image of an open surgical procedure can be obtained by amedical imaging device 96. The robotic hub 22 processes the images ofthe surgical site for subsequent display on the surgeon's console 18 orthe AR device 66 worn by the surgeon, or other person in the surgicaloperating room 16.

The optical components of the imaging device 24, 96 or AR device 66 mayinclude one or more illumination sources and/or one or more lenses. Theone or more illumination sources may be directed to illuminate portionsof the surgical field. One or more image sensors may receive lightreflected or refracted from tissue and instruments in the surgicalfield.

In various aspects, the imaging device 24 is configured for use in aminimally invasive surgical procedure. Examples of imaging devicessuitable for use with this disclosure include, but not limited to, anarthroscope, angioscope, bronchoscope, choledochoscope, colonoscope,cytoscope, duodenoscope, enteroscope, esophagogastro-duodenoscope(gastroscope), endoscope, laryngoscope, nasopharyngo-neproscope,sigmoidoscope, thoracoscope, and ureteroscope. In various aspects, theimaging device 96 is configured for use in an open (invasive) surgicalprocedure.

In various aspects, the visualization system 8 includes one or moreimaging sensors, one or more image-processing units, one or more storagearrays, and one or more displays that are strategically arranged withrespect to the sterile field. In one aspect, the visualization system 8includes an interface for HL7, PACS, and EMR. In one aspect, the imagingdevice 24 may employ multi-spectrum monitoring to discriminatetopography and underlying structures. A multi-spectral image capturesimage data within specific wavelength ranges in the electromagneticspectrum. Wavelengths are separated by filters or instruments sensitiveto particular wavelengths, including light from frequencies beyond thevisible light range, e.g., IR and ultraviolet. Spectral imaging canextract information not visible to the human eye. Multi-spectrummonitoring can relocate a surgical field after a surgical task iscompleted to perform tests on the treated tissue.

FIG. 2 depicts a primary display 19 positioned in the sterile field tobe visible to an operator at the operating table 14. A visualizationtower 11 is positioned outside the sterile field and includes a firstnon-sterile display 7 and a second non-sterile display 9, which faceaway from each other. The visualization system 8, guided by the hub 6,is configured to utilize the displays 7, 9, 19 to coordinate informationflow to operators inside and outside the sterile field. For example, thehub 6 may cause the visualization system 8 to display AR images of thesurgical site, as recorded by an imaging device 24, 96 on a non-steriledisplay 7, 9, or through the AR device 66, while maintaining a live feedof the surgical site on the primary display 19 or the AR device 66. Thenon-sterile display 7, 9 can permit a non-sterile operator to perform adiagnostic step relevant to the surgical procedure, for example.

FIG. 3 depicts a hub 6 in communication with a visualization system 8, arobotic system 10, and a handheld intelligent surgical instrument 12.The hub 6 includes a hub display 35, an imaging module 38, a generatormodule 40, a communication module 30, a processor module 32, a storagearray 34, and an operating room mapping module 33. The hub 6 furtherincludes a smoke evacuation module 26 and/or a suction/irrigation module28. In various aspects, the imaging module 38 comprises an AR device 66and the processor module 32 comprises an integrated video processor andan augmented reality modeler (e.g., as shown in FIG. 10). A modularlight source may be adapted for use with various imaging devices. Invarious examples, multiple imaging devices may be placed at differentpositions in the surgical field to provide multiple views (e.g.,non-invasive, minimally invasive, invasive or open surgical procedures).The imaging module 38 can be configured to switch between the imagingdevices to provide an optimal view. In various aspects, the imagingmodule 38 can be configured to integrate the images from the differentimaging devices and provide an augmented reality interactive experienceduring a surgical procedure as described herein.

FIG. 4 shows a surgical data network 51 comprising a modularcommunication hub 53 configured to connect modular devices located inone or more operating theaters/rooms of a healthcare facility to acloud-based system. The cloud 54 may include a remote server 63 (FIG. 5)coupled to a storage device 55. The modular communication hub 53comprises a network hub 57 and/or a network switch 59 in communicationwith a network router 61. The modular communication hub 53 is coupled toa local computer system 60 to process data. Modular devices 1 a-1 n inthe operating theater may be coupled to the modular communication hub53. The network hub 57 and/or the network switch 59 may be coupled to anetwork router 61 to connect the devices 1 a-1 n to the cloud 54 or thelocal computer system 60. Data associated with the devices 1 a-1 n maybe transferred to cloud-based computers via the router for remote dataprocessing and manipulation. The operating theater devices 1 a-1 n maybe connected to the modular communication hub 53 over a wired channel ora wireless channel. The surgical data network 51 environment may beemployed to provide an augmented reality interactive experience during asurgical procedure as described herein and in particular providingaugmented images if the surgical field to one or more than one remotedisplay 58.

FIG. 5 illustrates a computer-implemented interactive surgical system50. The computer-implemented interactive surgical system 50 is similarin many respects to the computer-implemented interactive surgical system1. The computer-implemented interactive surgical system 50 includes oneor more surgical systems 52, which are similar in many respects to thesurgical systems 2. Each surgical system 52 includes at least onesurgical hub 56 in communication with a cloud 54 that may include aremote server 63. In one aspect, the computer-implemented interactivesurgical system 50 comprises a modular control tower 23 connected tomultiple operating theater devices such as, for example, intelligentsurgical instruments, robots, and other computerized devices located inthe operating theater. As shown in FIG. 6, the modular control tower 23comprises a modular communication hub 53 coupled to a computer system60.

Back to FIG. 5, the modular control tower 23 is coupled to an imagingmodule 38 that is coupled to an endoscope 98, a generator module 27 thatis coupled to an energy device 99, a smoke evacuator module 76, asuction/irrigation module 78, a communication module 13, a processormodule 15, a storage array 16, a smart device/instrument 21 optionallycoupled to a display 39, and a sensor module 29. The operating theaterdevices are coupled to cloud computing resources such as server 63, datastorage 55, and displays 58 via the modular control tower 23. A robothub 72 also may be connected to the modular control tower 23 and to theservers 63, data storage 55, and displays 58. The devices/instruments21, visualization systems 58, among others, may be coupled to themodular control tower 23 via wired or wireless communication standardsor protocols, as described herein. The modular control tower 23 may becoupled to a hub display 65 (e.g., monitor, screen) to display augmentedimages received comprising overlaid virtual objects on the real surgicalfield received from the imaging module 38, device/instrument display 39,and/or other visualization systems 58. The hub display 65 also maydisplay data received from devices connected to the modular controltower 23 in conjunction with images and overlaid images.

FIG. 6 illustrates a surgical hub 56 comprising a plurality of modulescoupled to the modular control tower 23. The modular control tower 23comprises a modular communication hub 53, e.g., a network connectivitydevice, and a computer system 60 to provide local processing,visualization, and imaging of augmented surgical information, forexample. The modular communication hub 53 may be connected in a tieredconfiguration to expand the number of modules (e.g., devices) that maybe connected to the modular communication hub 53 and transfer dataassociated with the modules to the computer system 60, cloud computingresources, or both. Each of the network hubs/switches 57, 59 in themodular communication hub 53 may include three downstream ports and oneupstream port. The upstream network hub/switch 57, 59 is connected to aprocessor 31 to provide a communication connection to the cloudcomputing resources and a local display 67. Communication to the cloud54 may be made either through a wired or a wireless communicationchannel.

The computer system 60 comprises a processor 31 and a network interface37. The processor 31 is coupled to a communication module 41, storage45, memory 46, non-volatile memory 47, and input/output interface 48 viaa system bus. The system bus can be any of several types of busstructure(s) including the memory bus or memory controller, a peripheralbus or external bus, and/or a local bus using any variety of availablebus architectures.

The processor 31 comprises an augmented reality modeler (e.g., as shownin FIG. 10) and may be implemented as a single-core or multicoreprocessor such as those known under the trade name ARM Cortex by TexasInstruments. In one aspect, the processor may be an LM4F230H5QR ARMCortex-M4F Processor Core, available from Texas Instruments, forexample, comprising an on-chip memory of 256 KB single-cycle flashmemory, or other non-volatile memory, up to 40 MHz, a prefetch buffer toimprove performance above 40 MHz, a 32 KB single-cycle serial randomaccess memory (SRAM), an internal read-only memory (ROM) loaded withStellarisWare® software, a 2 KB electrically erasable programmableread-only memory (EEPROM), and/or one or more pulse width modulation(PWM) modules, one or more quadrature encoder inputs (QEI) analogs, oneor more 12-bit analog-to-digital converters (ADCs) with 12 analog inputchannels, details of which are available for the product datasheet.

The system memory includes volatile memory and non-volatile memory. Thebasic input/output system (BIOS), containing the basic routines totransfer information between elements within the computer system, suchas during start-up, is stored in non-volatile memory. For example, thenon-volatile memory can include ROM, programmable ROM (PROM),electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatilememory includes random-access memory (RAM), which acts as external cachememory. Moreover, RAM is available in many forms such as SRAM, dynamicRAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and directRambus RAM (DRRAM).

The computer system 60 also includes removable/non-removable,volatile/non-volatile computer storage media, such as for example diskstorage. The disk storage includes, but is not limited to, devices likea magnetic disk drive, floppy disk drive, tape drive, Jaz drive, Zipdrive, LS-60 drive, flash memory card, or memory stick. In addition, thedisk storage can include storage media separately or in combination withother storage media including, but not limited to, an optical disc drivesuch as a compact disc ROM device (CD-ROM), compact disc recordabledrive (CD-R Drive), compact disc rewritable drive (CD-RW Drive), or adigital versatile disc ROM drive (DVD-ROM). To facilitate the connectionof the disk storage devices to the system bus, a removable ornon-removable interface may be employed.

In various aspects, the computer system 60 of FIG. 6, the imaging module38 and/or visualization system 58, and/or the processor module 15 ofFIGS. 4-6, may comprise an image processor, image-processing engine,graphics processing unit (GPU), media processor, or any specializeddigital signal processor (DSP) used for the processing of digitalimages. The image processor may employ parallel computing with singleinstruction, multiple data (SIMD) or multiple instruction, multiple data(MIMD) technologies to increase speed and efficiency. The digitalimage-processing engine can perform a range of tasks. The imageprocessor may be a system on a chip with multicore processorarchitecture.

FIG. 7 illustrates an augmented reality system 263 comprising anintermediate signal combiner 64 positioned in the communication pathbetween an imaging module 38 and a surgical hub display 67. The signalcombiner 64 combines audio and/or image data received from an imagingmodule 38 and/or an AR device 66. The surgical hub 56 receives thecombined data from the combiner 64 and overlays the data provided to thedisplay 67, where the overlaid data is displayed. The imaging device 68may be a digital video camera and the audio device 69 may be amicrophone. The signal combiner 64 may comprise a wireless heads-updisplay adapter to couple to the AR device 66 placed into thecommunication path of the display 67 to a console allowing the surgicalhub 56 to overlay data on the display 67.

FIG. 8 illustrates an augmented reality (AR) system comprising anintermediate signal combiner positioned in the communication pathbetween an imaging module and a surgical hub display. FIG. 8 illustratesan AR device 66 worn by a surgeon 73 to communicate data to the surgicalhub 56. Peripheral information of the AR device 66 does not includeactive video. Rather, the peripheral information includes only devicesettings, or signals that do not have same demands of refresh rates.Interaction may augment the surgeon's 73 information based on linkagewith preoperative computerized tomography (CT) or other data linked inthe surgical hub 56. The AR device 66 can identify structure—ask whetherinstrument is touching a nerve, vessel, or adhesion, for example. The ARdevice 66 may include pre-operative scan data, an optical view, tissueinterrogation properties acquired throughout procedure, and/orprocessing in the surgical hub 56 used to provide an answer. The surgeon73 can dictate notes to the AR device 66 to be saved with patient datain the hub storage 45 for later use in report or in follow up.

The AR device 66 worn by the surgeon 73 links to the surgical hub 56with audio and visual information to avoid the need for overlays, andallows customization of displayed information around periphery of view.The AR device 66 provides signals from devices (e.g., instruments),answers queries about device settings, or positional information linkedwith video to identify quadrant or position. The AR device 66 has audiocontrol and audio feedback from the AR device 66. The AR device 66 isable to interact with other systems in the operating theater and havefeedback and interaction available wherever the surgeon 73 is viewing.For example, the AR device 66 may receive voice or gesture initiatedcommands and queries from a surgeon, and the AR device 66 may providefeedback in the form of one or more modalities including audio, visual,or haptic touch.

FIG. 9 illustrates a surgeon 73 wearing an AR device 66, a patient 74,and may include a camera 96 in an operating room 75. The AR device 66worn by the surgeon 73 may be used to present to the surgeon 73 avirtual object overlaid on a real time image of the surgical fieldthrough augmented reality display 89 or through the hub connecteddisplay 67. The real time image may include a portion of a surgicalinstrument 77. The virtual object may not be visible to others withinthe operating room 75 (e.g., surgical assistant or nurse), though theyalso may wear AR devices 66. Even if another person is viewing theoperating room 75 with an AR device 66, the person may not be able tosee the virtual object or may be able to see the virtual object in ashared augmented reality with the surgeon 73, or may be able to see amodified version of the virtual object (e.g., according tocustomizations unique to the surgeon 73) or may see different virtualobjects.

A virtual object and/or data may be configured to appear on a portion ofa surgical instrument 77 or in a surgical field of view captured by animaging module 38, an imaging device 68 during minimally invasivesurgical procedures, and/or the camera 96 during open surgicalprocedures. In the illustrated example, the imaging module 38 is alaparoscopic camera that provides a live feed of a surgical area duringa minimally invasive surgical procedure. An AR system may presentvirtual objects that are fixed to a real object without regard to aperspective of a viewer or viewers of the AR system (e.g., the surgeon73). For example, a virtual object may be visible to a viewer of the ARsystem inside the operating room 75 and not visible to a viewer of theAR system outside the operating room 75. The virtual object may bedisplayed to the viewer outside the operating room 75 when the viewerenters the operating room 75. The augmented image may be displayed onthe surgical hub display 67 or the augmented reality display 89.

The AR device 66 may include one or more screens or lens, such as asingle screen or two screens (e.g., one per eye of a user). The screensmay allow light to pass through the screens such that aspects of thereal environment are visible while displaying the virtual object. Thevirtual object may be made visible to the surgeon 73 by projectinglight. A virtual object may appear to have a degree of transparency ormay be opaque (i.e., blocking aspects of the real environment).

An AR system may be viewable to one or more viewers, and may includedifferences among views available for the one or more viewers whileretaining some aspects as universal among the views. For example, aheads-up display may change between two views while virtual objectsand/or data may be fixed to a real object or area in both views. Aspectssuch as a color of an object, lighting, or other changes may be madeamong the views without changing a fixed position of at least onevirtual object.

A user may see a virtual object and/or data presented in an AR system asopaque or as including some level of transparency. In an example, theuser may interact with the virtual object, such as by moving the virtualobject from a first position to a second position. For example, the usermay move an object with his or her hand. This may be done in the ARsystem virtually by determining that the hand has moved into a positioncoincident or adjacent to the object (e.g., using one or more cameras,which may be mounted on the AR device 66, such as AR device camera 79 orseparate 96, and which may be static or may be controlled to move), andcausing the object to move in response. Virtual aspects may includevirtual representations of real world objects or may include visualeffects, such as lighting effects, etc. The AR system may include rulesto govern the behavior of virtual objects, such as subjecting a virtualobject to gravity or friction, or may include other predefined rulesthat defy real world physical constraints (e.g., floating objects,perpetual motion, etc.). The AR device 66 may include a camera 79 on theAR device 66 (not to be confused with the camera 96, separate from theAR device 66). The AR device camera 79 or the camera 96 may include aninfrared camera, an infrared filter, a visible light filter, a pluralityof cameras, a depth camera, etc. The AR device 66 may project virtualitems over a representation of a real environment, which may be viewedby a user.

The AR device 66 may be used in the operating room 75 during a surgicalprocedure, for example performed by the surgeon 73 on the patient 74.The AR device 66 may project or display virtual objects, such as avirtual object during the surgical procedure to augment the surgeon'svision. The surgeon 73 may view a virtual object using the AR device 66,a remote controller for the AR device 66, or may interact with a virtualobject, for example, using a hand to “interact” with a virtual object ora gesture recognized by the camera 79 of the AR device 66. A virtualobject may augment a surgical tool such as the surgical instrument 77.For example, the virtual object may appear (to the surgeon 73 viewingthe virtual object through the AR device 66) to be coupled with orremain a fixed distance from the surgical instrument 77. In anotherexample, the virtual object may be used to guide the surgical instrument77, and may appear to be fixed to the patient 74. In certain examples, avirtual object may react to movements of other virtual or real-worldobjects in the surgical field. For example, the virtual object may bealtered when a surgeon is manipulating a surgical instrument inproximity to the virtual object.

The augmented reality display system imaging device 38 capture a realimage of a surgical area during a surgical procedure. An augmentedreality display 89, 67 presents an overlay of an operational aspect ofthe surgical instrument 77 onto the real image of the surgical area. Thesurgical instrument 77 includes communications circuitry 231 tocommunicate operational aspects and functional data from the surgicalinstrument 77 to the AR device 66 via communication communicationscircuitry 233 on the AR device 66. Although the surgical instrument 77and the AR device 66 are shown in RF wireless communication betweencircuits 231, 233 as indicated by arrows B, C, other communicationtechniques may employed (e.g., wired, ultrasonic, infrared, etc.). Theoverlay is related to the operational aspect of the surgical instrument77 being actively visualized. The overlay combines aspects of tissueinteraction in the surgical area with functional data from the surgicalinstrument 77. A processor portion of the AR device 66 is configured toreceive the operational aspects and functional data from the surgicalinstrument 77, determine the overlay related to the operation of thesurgical instrument 77, and combine the aspect of the tissue in thesurgical area with the functional data from the surgical instrument 77.The augmented images indicate alerts relative to device performanceconsiderations, alerts of incompatible usage, alerts on incompletecapture. Incompatible usage includes tissue out range conditions andtissue incorrectly balanced within the jaws of the end effector.Additional augmented images provide an indication of collateral eventsincluding indication of tissue tension and indication of foreign objectdetection. Other augmented images indicate device status overlays andinstrument indication.

FIG. 10 illustrates a system 83 for augmenting images of a surgicalfield with information using an AR display 89, in accordance with atleast one aspect of this disclosure. The system 83 may be used toperform the techniques described hereinbelow, for example, by using theprocessor 85. The system 83 includes one aspect of an AR device 66 thatmay be in communication with a database 93. The AR device 66 includes aprocessor 85, memory 87, an AR display 89, and a camera 79. The ARdevice 66 may include a sensor 90, a speaker 91, and/or a hapticcontroller 92. The database 93 may include image storage 94 orpreoperative plan storage 95.

The processor 85 of the AR device 66 includes an augmented realitymodeler 86. The augmented reality modeler 86 may be used by theprocessor 85 to create the augmented reality environment. For example,the augmented reality modeler 86 may receive images of the instrument ina surgical field, such as from the camera 79 or sensor 90, and createthe augmented reality environment to fit within a display image of thesurgical field of view. In another example, physical objects and/or datemay be overlaid on the surgical field of view and/or the surgicalinstruments images and the augmented reality modeler 86 may use physicalobjects and data to present the augmented reality display of virtualobject s and/or data in the augmented reality environment. For example,the augmented reality modeler 86 may use or detect an instrument at asurgical site of the patient and present a virtual object and/or data onthe surgical instrument and/or an image of the surgical site in thesurgical field of view captured by the camera 79. The AR display 89 maydisplay the AR environment overlaid on a real environment. The display89 may show a virtual object and/or data, using the AR device 66, suchas in a fixed position in the AR environment.

The AR device 66 may include a sensor 90, such as an infrared sensor.The camera 79 or the sensor 90 may be used to detect movement, such as agesture by a surgeon or other user, that may be interpreted by theprocessor 85 as attempted or intended interaction by the user with thevirtual target. The processor 85 may identify an object in a realenvironment, such as through processing information received using thecamera 79. In other aspects, the sensor 90 may be a tactile, audible,chemical, or thermal sensor to generate corresponding signals that maycombined with various data feeds to create the augmented environment.The sensor 90 may include binaural audio sensors (spatial sound),inertial measurement (accelerometer, gyroscope, magnetometer) sensors,environmental sensors, depth camera sensors, hand and eye trackingsensors, and voice command recognition functions.

The AR display 89, for example during a surgical procedure, may present,such as within a surgical field while permitting the surgical field tobe viewed through the AR display 89, a virtual feature corresponding toa physical feature hidden by an anatomical aspect of a patient. Thevirtual feature may have a virtual position or orientation correspondingto a first physical position or orientation of the physical feature. Inan example, the virtual position or orientation of the virtual featuremay include an offset from the first physical position or orientation ofthe physical feature. The offset may include a predetermined distancefrom the augmented reality display, a relative distance from theaugmented reality display to the anatomical aspect, or the like.

In one example, the AR device 66 may be an individual AR device. In oneaspect, the AR device 66 may be a HoloLens 2 AR device manufactured byMicrosoft of Redmond, Wash. This AR device 66 includes a visor withlenses and binaural audio features (spatial sound), inertial measurement(accelerometer, gyroscope, magnetometer), environmental sensors, depthcamera, and video camera, hand and eye tracking, and voice commandrecognition functions. It provides an improved field of view with highresolution by using mirrors to direct waveguides in front of wearer'seyes. Images can be enlarged by changing angles of mirrors. It alsoprovides eye tracking to recognize users and adjust lens widths forspecific users.

In another example, the AR device 66 may be a Snapchat Spectacles 3 ARdevice. This AR device provides the ability to capture paired images andrecreate 3D depth mapping, add in virtual effects, and replay 3D videos.The AR device includes two HD cameras to capture 3D photos and videos at60 fps—while four built-in microphones record immersive, high-fidelityaudio. Images from both cameras combine to build out a geometric map ofthe real world around the user to provide a new sense of depthperception. Photos and videos may be wirelessly synchronized to externaldisplay devices.

In yet another example, the AR device 66 may be a Glass 2 AR device byGoogle. This AR device provides inertial measurement (accelerometer,gyroscope, magnetometer) information overlaid on lens (out of view) tosupplement information.

In another example, the AR device 66 may be an Echo Frames AR device byAmazon. This AR device does not have cameras/displays. A microphone andspeaker are linked to Alexa. This AR device provides less functionalitythan a heads-up display.

In yet another example, the AR device 66 may be a Focals AR device byNorth (Google). This AR device provides notification pusher/smartwatchanalog; inertial measurement, screen overlay of information (weather,calendar, messages), voice control (Alexa) integration. This AR deviceprovides basic heads-up display functionality.

In another example, the AR device 66 may be an Nreal AR device. This ARdevice includes spatial sound, two environmental cameras, a photocamera, IMU (accelerometer, gyroscope), ambient light sensor, proximitysensor functionality. A nebula projects application information onlenses.

In various other examples, the AR device 66 may be any one of thefollowing commercially available AR devices: Magic Leap 1, EpsonMoverio, Vuzix Blade AR, ZenFone AR, Microsoft AR glasses prototype,EyeTap to create collinear light to that of the environment directlyinto the retina. A beam splitter makes the same light seen by the eyeavailable to the computer to process and overlay information, forexample. AR visualization systems include HUD, contact lenses , glasses,virtual reality (VR) headsets, virtual retinal display, on in operatingroom displays, and/or smart contact lenses (bionic lenses).

Multi-user interfaces for the AR device 66 include virtual retinaldisplays such as raster displays drawn directly on retinas instead of ona screen in front of the eye, smart televisions, smart phones, and/orspatial displays such as Sony spatial display systems.

Other AR technology may include, for example, AR capture devices andsoftware applications, AR creation devices and software applications,and AR cloud devices and software applications. AR capture devices andsoftware applications include, for example, Apple Polycam app, Ubiquity6 (Mirrorworld using Display.land app)—users can scan and get 3d imageof real world (to create 3D model). AR creation devices and softwareapplications include, for example, Adobe Aero, Vuforia, ARToolKit,Google ARCore, Apple ARKit, MAXST, Aurasma, Zappar, Blippar. AR clouddevices and software applications include, for example, Facebook, Google(world geometry, objection recognition, predictive data), Amazon ARCloud (commerce), Microsoft Azure, Samsung Project Whare, Niantic, MagicLeap.

One aspect of the following disclosure describes various overlays ofsurgical instrument operational aspects or functions onto a live videostream of a surgical area as visualized through a laparoscopic camerasurgical field of view during a minimally invasive surgical procedure.The overlay is related to the operation of one of the surgicalinstruments or devices being actively visualized. The overlays combineaspects of tissue/organ interaction with functional data received fromsurgical instruments used in the surgical procedure. Surgicalinstruments may include graspers, clamps, staplers, ultrasonic, RF, orcombination of each of these instruments. In regard to graspers andclamps, aspects of tissue parameters may include incomplete capture ofthe tissue along with the status of the clamp or magnitude of the clamp.In regard to a surgical stapler, aspects of tissue parameters mayinclude tissue capture location, tissue compression, clamping, or firingsufficiency of a surgical stapler. In regard to advanced energy devices,such ultrasonic or RF devices, aspects of tissue parameters may includeimpedance, cautery status, bleeding magnitude, and aspects of instrumentfunction may include energy level, timing, clamp pressure, among others,for examples. The augmented images shown in FIGS. 11-35 hereinbelow maybe viewed on a local display, a remote display, and/or an AR device asdescribed hereinabove in connection with FIGS. 1-10. Although theaugmented images are described as being visualized through alaparoscopic camera during a minimally invasive surgical procedure, theimages may be captured during non-invasive and invasive (e.g., open)surgical procedures without limiting the scope of this disclosure inthis context. These aspects are described hereinbelow.

FIGS. 11-75 describe various augmented images visualized through alaparoscopic camera during a minimally invasive surgical procedure. Anaugmented reality display system is used during a surgical procedure.The augmented reality display system comprises an imaging device tocapture a real image of a surgical area during the surgical procedure,an augmented reality display to present an overlay of an operationalaspect of a surgical instrument onto the real image of the surgicalarea, and a processor. The overlay is related to the operational aspectof the surgical instrument being actively visualized. The overlaycombines aspects of tissue interaction in the surgical area withfunctional data from the surgical instrument. The processor isconfigured to receive the functional data for the surgical instrument,determine the overlay related to the operation of the surgicalinstrument, and combine the aspect of the tissue in the surgical areawith the functional data from the surgical instrument. The augmentedimages indicate alerts relative to device performance considerations,alerts of incompatible usage, alerts on incomplete capture. Incompatibleusage includes tissue out range conditions and tissue incorrectlybalanced within the jaws of the end effector. Additional augmentedimages provide an indication of collateral events including indicationof tissue tension and indication of foreign object detection. Otheraugmented images indicate device status overlays and instrumentindication.

FIGS. 11-75 also describe functional overlays of instrument criticaloperations or parameters to clearly represent the surgical stapler,energy device, or aspects of its interaction. In one aspect, theoverlaid data is adjusted by an aspect detected by the surgical hub tomodify the overlay from the information merely detected by the sourceinstrument to add context. In another aspect, the displays may beadjusted or modified by the user and as a result also result inmodifications of the surgical instrument being monitored operation.

FIG. 11 is an augmented image 100 of a live feed of a surgical area 118as visualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating appropriate tissue 112 captured betweenjaws 110 of a surgical instrument end effector 108. The operationalaspect of the surgical instrument is clamping the tissue 112 between thejaws 110 of the end effector 108. The aspect of the tissue 112 isappropriate capture of the tissue 112 between the jaws 110 of the endeffector 108. The laparoscopic field of view 102 of the surgical area118 shows a surgical instrument end effector 108 grasping tissue 112 inits jaws 110. The augmented image 100 shows a virtual graphical alertoverlay 106 superimposed on the anvil 110 of the end effector 108. Thevirtual superimposed graphical alert overlay 106 indicates the amount oftissue grasped in the end effector 108 jaws 110 to inform the OR team ofstaple cartridge reload selection for stapling, overloading tissue forenergy, etc. A first superimposed alert 104 informs that the tissue 112grasped with the jaws 110 at the proximal end of the end effector 108 isout of range. A second superimposed alert 116 informs that the tissue112 grasped with the jaws 110 at the medial portion of the end effector108 is within reload range. A third superimposed alert 114 informs thatthe tissue 112 grasped with the jaws at the distal end of the endeffector 108 is over the cut line. The superimposed graphical alertoverlay 106 apply to energy based surgical instruments, stapler basedsurgical instruments, manipulation tools, and the like.

FIG. 12 is an augmented image 200 of a live feed of a surgical area 212as visualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating tissue 206 position proximally capturedbetween jaws 210 of a surgical instrument end effector 204 as a tissueaspect. Aspects of the tissue include incorrectly captured or positionedtissue 206 between the jaws 210 of the end effector 204. Thelaparoscopic field of view 202 of the surgical area 212 shows thesurgical instrument end effector 204 grasping tissue 206 between thejaws 210 of the end effector 204. The augmented image 200 shows agraphical alert overlay 208 superimposed on one of the jaws 210 of theend effector 204 to indicate that the tissue 206 is grasped excessivelyproximally 206 with the jaws 210 of the end effector 204. Although thesuperimposed alert 208 applies primarily to energy based surgicalinstruments, for example, similar alerts may be superimposed in surgicalstapler instruments.

FIG. 13 is an augmented image 300 of a live feed of a surgical area 324as visualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating tissue 322 insufficiently captured betweenthe jaws 318 of a surgical instrument end effector 320 as a tissueaspect. The laparoscopic field of view 302 of the surgical area 324shows a surgical instrument end effector 320 grasping tissue 322 withthe jaws 318 of the end effector 320. The augmented image 300 shows agraphical alert overlay 304 superimposed on the image of the surgicalarea 324 to indicate insufficiently captured tissue 322 relative to theend of cut in the jaws 318 of the end effector 320.

The augmented image 300 also comprises a first sub image 308 showing agraphic image 306 of the general anatomy superimposed on or adjacent tothe surgical field of view 302 and a reference frame 310 of the actualanatomy superimposed on or adjacent to the surgical field of view 302.The augmented image 300 also comprises a second sub image 312 showingthe type of surgical instrument in use, the energy level if applicable,and the current surgical procedure. The second sub image 312 may besuperimposed on or located adjacent to the surgical field of view 302.The augmented image 300 shows an ultrasonic surgical instrument beingused in a surgical procedure at an energy level set to 5 Max to achieveadvanced hemostasis. A graphic image 316 of the surgical instrument isshown superimposed on a graphic image 314 of the incomplete tissuecapture alert overlay 304. Accordingly, the augmented image 300 providesseveral virtual objects that inform the OR team of insufficientlycaptured tissue 322 relative to the end of cut. The superimposedincomplete tissue capture alert overlay 304 applies to energy basedsurgical instruments as well as surgical stapler instruments, and thelike.

FIG. 14 is another augmented image 330 of a live feed of a surgical area359 as visualized through a laparoscopic camera during a minimallyinvasive surgical procedure indicating tissue 348 insufficientlycaptured between jaws 358 of a surgical instrument end effector 354 as atissue aspect. The laparoscopic field of view 332 of the surgical area359 shows a surgical instrument end effector 354 with insufficientlycaptured tissue 352 relative to the end of cut in the jaws 359, 358 ofthe end effector 354. The augmented image 330 shows a graphical alertoverlay 352 superimposed on the tissue 348 to indicate incomplete tissuecapture. In the illustrated example, a jaw 358, e.g., anvil, of the endeffector 354 includes a series of light emitting diode (LED) indicators356 to indicate the position of tissue 352 grasped with the jaws 350,358 of the end effector 354. Also shown in the surgical field of view332 are two rows of staples 334 in a previously stapled portion oftissue 352.

The augmented image 330 comprises a first sub image 338 showing agraphic image 336 of the general anatomy illustrated in the laparoscopicfield of view 332 and a reference frame 340 of the actual anatomy shownin the laparoscopic field of view 332. The augmented image 330 comprisesa second sub image 342 showing the type of instrument being used and thesurgical procedure. In the illustrated example, a powered vascularsurgical stapler is being used in a vascular surgical procedure. Alsoshown in the second sub image 342 is a graphic image of the staplercartridge 346 of the powered surgical stapler and a graphic 344superimposed on the graphic image of the stapler cartridge 346 toindicate the cut line of the powered surgical stapler.

FIG. 15 is another augmented image 360 of a live feed of a surgical area374 as visualized through a laparoscopic camera during a minimallyinvasive surgical procedure indicating insufficiently captured tissue372 between jaws 370 of a surgical instrument end effector 368 as atissue aspect. The laparoscopic field of view 362 of a surgical area 374shows a surgical instrument end effector 368 with insufficientlycaptured tissue 372 relative to the end of cut in the jaws 370 of theend effector 368. The augmented image 360 shows a graphical alertoverlay 366 superimposed on the anvil of the end effector 368 toindicate incomplete tissue capture. A first superimposed alert 364informs that the tissue 372 grasped at the proximal end of the jaws 370of the end effector 368 is out of range. A second superimposed alert 378informs that the tissue 372 grasped at the medial portion of the jaws370 of the end effector 368 is within reload range. A third superimposedalert 376 informs that the tissue 372 grasped at the distal end of thejaws 370 in the end effector 368 is over the cut line.

FIG. 16 is an augmented image 400 of a live feed of a surgical area 422as visualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating tissue 420 tension as a tissue aspect. Thelaparoscopic field of view 402 shows tissue 420 grasped between the jaws424 of a surgical instrument end effector 418. The augmented image 400shows a graphical alert overlay 404 superimposed in the surgical area422 and a coloration of the tissue 420 to warn that the grasped tissue420 is under tension. The superimposed tissue tension alert 404 andcoloration of the tissue 420 grasped with the jaws 424 of the endeffector 418 informs that the grasped tissue 420 is under tension. Thesuperimposed tissue tension alert overlay 404 and coloration of tissue420 under tension applies to energy based surgical instruments, surgicalstapler instruments, as well as manipulation tools.

The augmented image 400 comprises a first sub image 408 showing agraphic image 406 of the general anatomy illustrated in the laparoscopicfield of view 402 and a reference frame 410 of the actual anatomy shownin the laparoscopic field of view 402. The augmented image 400 alsocomprises a second sub image 412 showing the type of surgical instrumentin use, the energy level being applied, if applicable, and the currentsurgical procedure. The augmented image 400 shows an ultrasonic surgicalinstrument being used in a surgical procedure at an energy level of 5Max to achieve advanced hemostasis. A graphic image 416 of the surgicalinstrument is shown superimposed on a graphic image 414 of the reducetension alert overlay 404. Accordingly, the augmented image 400 providesseveral options for informing the OR team to reduce the tension of thecaptured tissue 420 relative to the end of cut. The superimposed reducetension alert overlay 404 applies to energy based surgical instrumentsas well as surgical stapler instruments, and the like.

FIG. 17 is another augmented image 430 of a live feed of a surgical area442 as visualized through a laparoscopic camera during a minimallyinvasive surgical procedure indicating tissue 444 tension as a tissueaspect. The laparoscopic field of view 432 shows a surgical instrumentend effector 436 grasping tissue 444 between the jaws 438 of the endeffector 436. In the augmented image 430, coloration is added to thegrasped tissue 444 to indicate that the grasped tissue 444 is undertension. Tissue 444 grasped with the jaws 438 of the end effector 436 iscolored to indicate that the tissue 444 is under tension. Also shown inthe laparoscopic field of view 432 are rows of previously appliedstaples 434. Also shown is a set of LEDs 440 to indicate the position ofthe tissue 444 grasped with the jaws 438 of the end effector 436.

FIG. 18 is a plurality of graphic images 500 indicating jaw closureposition as an operational aspect of a surgical instrument as shown inFIGS. 11-17. The plurality of graphic images 500 comprise staticgraphics and alerts overlaid on the image of the surgical area and/oroverlaid on the jaw of the end effector itself. The superimposed graphicimages 500 apply to energy based surgical instruments, stapler basedsurgical instruments, manipulation tools, and the like.

A first image 502 comprises a first graphical overlay 504 showing thePatient Information, Procedure, and Case ID. A second graphical overlay506 informs the type of surgical stapler instrument under use in thesurgical procedure, e.g., a surgical stapler with a closed staple heightof 1.5 mm as shown. A third graphical overlay 508 informs of the degreeof articulation of the surgical stapler A fourth graphical overlay 510is a pop-up error display. Finally, a fifth graphical overlay 512informs of the energy surgical instrument under use in the surgicalprocedure, e.g., ultrasonic instrument operating between energy levelsof 3 Min to 5 Max.

A second image 514 includes all of the graphical overlays 506, 508, 510,512 explained in the description of the first image 502 with theaddition of a sixth graphical overlay 516 that informs of the jaw of thesurgical stapler being closed.

A third image 518 includes all of the graphical overlays 506, 508, 510,512 explained in the description of the first image 502 with theaddition of a sixth graphical overlay 520 that informs of the jaw of thesurgical stapler being partially closed.

A fourth image 522 includes all of the graphical overlays 506, 508, 510,512 explained in the description of the first image 502 with theaddition of a sixth graphical overlay 524 that informs of the jaw of thesurgical stapler being open.

FIG. 19 is an augmented image 530 of a live feed of a surgical area 544as visualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating jaw 536, 542 closure position as anoperational aspect of a surgical instrument. The laparoscopic field ofview 532 shows a partially closed surgical instrument end effector 540with tissue 534 grasped between the jaws 536, 542 of the end effector540 and a tissue manipulation tool 546 used to assist the tissuegrasping process. The augmented image 530 shows a graphical alertoverlay 538 superimposed on the end effector 540 indicating partialclosure.

FIG. 20 is an augmented image 550 of a live feed of the surgical area544 shown in FIG. 19 showing a fully closed surgical instrument endeffector 540 and a graphical alert overlay 546 showing jaw closedposition superimposed on the end effector 540. The laparoscopic field ofview of the surgical area 544 shows tissue 534 grasped in the jaws 536,542 of the end effector 540.

FIG. 21 is an augmented image 600 of a live feed of a surgical area 610as visualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating clamping on metal or foreign object as anoperational aspect of a surgical instrument. The laparoscopic field ofview 602 of the surgical area 610 shows the surgical instrument endeffector 604 clamped on a metal or foreign object embedded in the tissue608. The augmented image 600 shows a graphical alert overlay 614superimposed on a graphical overlay 612 of an upper jaw 606 of the endeffector 604 showing a foreign object clamped between the jaws of theend effector 604.

FIG. 22 is an augmented image 700 of a live feed of a surgical area 714as visualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating device residual heat warning whereoverheating is an operational aspect of a surgical instrument. Thelaparoscopic field of view of the surgical area 714 shows a surgicalinstrument end effector 706 with tissue 710 clamped between anultrasonic blade 710 and a clamp arm 704. The end effector 706 shows ahot ultrasonic blade 710. The augmented image 700 includes a graphicalalert overlay 708 superimposed on the hot ultrasonic blade 710. Thegraphical alert overlay 708 warns the user that the ultrasonic blade 710is building up residual heat, which will be transferred to thesurrounding tissue 712 captured between the ultrasonic blade 710 and aclamp arm 704 pivotally coupled thereto.

FIG. 23 is an augmented image 800 of a live feed of a surgical area 822as visualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating tissue 828 movement and flow as a tissueaspect. The laparoscopic view of the surgical area 822 shows themovement and flow of tissue 828 relative to tissue graspers 804, 824 anda surgical instrument end effector 812 with the tissue 828 grasped inthe jaws 824 of the end effector 812 and the jaws of the tissue graspers804, 824. The augmented image 800 shows graphical images 814, 820overlaid on surgical area 822. The first graphical image 814 includes anicon 816 of a surgical stapler and text identifying the surgicalinstrument as a “stapler.” The second image 820 includes an icon of agrasper 818. A sub image 810 alerts the OR team (e.g., the surgeon) ofpotential tissue movement over the cutline 808 displayed over an outlineof the stapler cartridge 806.

FIG. 24 is an augmented image 900 of a live feed of a surgical area 918as visualized through a laparoscopic camera during a minimally invasivesurgical procedure indicating the geometric relationship of thetransections to tissue 912 and other firings as a tissue aspect. Thelaparoscopic view of the surgical area 918 shows a surgical instrumentend effector 910 and the geometric relationships of transections to thetissue 912 grasped between the jaws 914, 916 of the end effector 910 andto other firings. The augmented image 900 shows graphical imagesoverlaid on the surgical area 918. A sub image 908 includes graphicsthat predict a complete transection with two staple lines showngraphically as active placement staple line 904 and complete staple line906.

FIG. 25 is an image 1000 indicating anvil orientation communication asan operation as an operational aspect of a surgical instrument. A firstimage 1002 shows all of the graphical overlays 504, 506, 508, 510, 512of the explained in the description of the first image 502 shown in FIG.18 with the addition of an anvil up/down indicator graphic 1004. In theillustrated example, the up/down indicator graphic 1004 shows that theanvil is in the up (e.g., open) position. Another image 1006 illustratesthe anatomy. A second image 1008 shows all of the graphical overlays504, 506, 508, 510, 512 of the explained in the description of the firstimage 502 shown in FIG. 18 and the image 1006 of the anatomy with theaddition of an anvil up/down indicator graphic 1010. In the illustratedexample, the up/down indicator graphic 1010 shows that the anvil is inthe down (e.g., closed) position.

FIG. 26 is an image 1100 indicating detected tissue thickness in thejaws of a surgical instrument end effector where tissue thickness is atissue aspect. The image 1100 shows a graphic screen 1102 with text 1114that tissue thickness is outside of range of reload with arecommendation to consider changing staple cartridge ad a warning icon1112. The graphic screen 1102 shows a detected thickness marker 1104positioned along a scale 1106 that shows detected tissue thickness. Thegraphic screen 1102 also shows the type of surgical instrument in use.The type of surgical instrument in use is a surgical stapler 1110 with aclosed staple height 1108 of 1.5 mm.

FIG. 27 is an image 1200 of temperature gradient display graphics 1202,1204, 1206 for an ultrasonic instrument. The first, second, and thirddisplay graphic 1202, 1204, 1206 may be overlaid on a surgical area of asurgical field of view displayed on a display as described in FIGS.11-26, informs of the instrument type, minimum and maximum output energylevels, and displays temperature gradient elements 1208, 1210, 1212 on aportion of the display graphic 1202, 1204, 1206. In the first displaygraphic 1202 the ultrasonic instrument is either off or the ultrasonicblade temperature is running at a normal temperature and a “green”gradient bar is not displayed or barely visible. In the second displaygraphic 1204 a temperature gradient element 1210 “heat map bar” rangingfrom green to yellow shows that the ultrasonic blade is approaching atemperature above normal. In the third display graphic 1206 atemperature gradient element 1212 “heat map bar” ranging from green tored shows that the ultrasonic blade temperature is above normal in anoverheated state.

FIG. 28 is an image 1300 of temperature icon display graphics 1302,1304, 1306 for an ultrasonic instrument. The first, second, and thirddisplay graphics 1302, 1304, 1306 may be overlaid on a surgical area ofa surgical field of view displayed on a display as described in FIGS.11-26, informs of the instrument type, minimum and maximum output energylevels, and displays temperature green, yellow, and red icons 1308,1310, 1310 on a portion of the display graphic 1302, 1304, 1306. In thefirst display graphic 1302 the ultrasonic instrument is operating in anormal condition and the temperature of the ultrasonic blade is in anormal range as shown by green icon 1308 and also shown by the greensemicircular temperature meter element 1314. In the second displaygraphic 1304 the yellow temperature icon 1310 shows that the ultrasonicblade is approaching a temperature above normal as also shown by theyellow semicircular temperature meter element 1316. In the third displaygraphic 1306 a red temperature icon 1312 shows that the ultrasonic bladetemperature is above normal and is in a overheated state as shown by thered semicircular temperature meter element 1318.

FIG. 29 is an image 1400 of ultrasonic blade temperature graphicelements 1410, 1412, 1414 mapped to an end effector 1418 jaw 1420position. The first, second, and third display graphic elements 1410,1412, 1414 may be overlaid on the jaw 1420 of an ultrasonic end effector1418 in a surgical area of a surgical field of view displayed on adisplay as described in FIGS. 11-26. The first temperature graphicelement 1410 shows that the ultrasonic blade is operating in a normalrange as shown by the green temperature graphic element 1410. The secondtemperature graphic element 1412 shows that the ultrasonic blade isapproaching a heated temperature state as shown by the yellowtemperature graphic element 1412. The third temperature graphic element1414 shows that the ultrasonic blade is in an overheated temperaturestate as shown by the red temperature graphic element 1414.

FIG. 30 is an image 1500 of an ultrasonic generator power level displaygraphic 1502. The display graphic 1502 informs of the generator type andminimum and maximum power levels. One or more graphic elements of theimage 1500 may be overlaid on the jaw of the ultrasonic end effector oranywhere in the surgical area in the surgical field of view displayed ona display as described in FIGS. 11-26.

FIG. 31 is an image 1600 of an ultrasonic generator power level displaygraphic 1502 with a pop-up warning graphic 1602 indicating that theultrasonic end effector jaw is overstuffed. The pop-up warning graphic1602 may include the language to show that too much tissue is in the jawand may result in an incomplete transection. One or more graphicelements of the image 1600 may be overlaid on the jaw of the ultrasonicend effector or anywhere in the surgical area in the surgical field ofview displayed on a display as described in FIGS. 11-26.

FIG. 32 is an image 1700 of an ultrasonic generator power level displaygraphic 1502 with a pop-up warning graphic 1702 indicating ultrasonicend effector jaw heat. The pop-up warning graphic 1702 may include thelanguage to show that the instrument temperature is hot and in oneaspect, the word hot may be shown in red. One or more graphic elementsof the image 1700 may be overlaid on the jaw of the ultrasonic endeffector or anywhere in the surgical area in the surgical field of viewdisplayed on a display as described in FIGS. 11-26.

FIG. 33 is an image 1800 of an electrosurgical generator display graphic1802 with a pop-up warning graphic 1802 indicating electrosurgical sealquality prediction. The pop-up warning graphic 1804 may include thelanguage Warning: Incomplete Seal and the electrosurgical generatordisplay graphic 1802 may include the language Error to show that anincomplete seal is predicted. One or more graphic elements of the image1800 may be overlaid on the jaw of the electrosurgical end effector oranywhere in the surgical area in the surgical field of view displayed ona display as described in FIGS. 11-26.

FIG. 34 is an image 1900 of a surgical stapler reload feedback. Theimage 1900 comprises a first display graphic 1902 indicating a suitabletissue thickness range for the surgical stapler in use. In theillustrated example, the suitable surgical stapler tissue thicknessrange is 1.5 mm-2.4 mm. A second display graphic 1904 shows the detectedtissue thickness. In the illustrated example, the detected tissuethickness is 2.6 mm. A third pop-up display graphic 1906 shows that thetissue thickness is 2.6 mm and is thicker than the selected reload andprovides a suggestion to consider a reload in the range of 2.0 mm-3.3mm. The image 1800 may be overlaid on the jaw of the electrosurgical endeffector or anywhere in the surgical area in the surgical field of viewdisplayed on a monitor or screen. One or more graphic elements of theimage 1900 may be overlaid on the jaw/anvil of the surgical stapler endeffector or anywhere in the surgical area in the surgical field of viewdisplayed on a display as described in FIGS. 11-26.

FIG. 35 is an image 2000 of a surgical stapler precompression countdown.The image 2000 comprises a first display graphic 1902 as shown in FIG.34 indicating the surgical stapler tissue thickness range of 1.5 mm-2.4mm. A second pop-up display graphic 2002 shows a surgical staplerprecompression countdown element to show the time remaining in theprecompression phase of the surgical stapler operation. One or moregraphic elements of the image 2000 may be overlaid on the jaw/anvil ofthe surgical stapler end effector or anywhere in the surgical area inthe surgical field of view displayed on a display as described in FIGS.11-26.

The following description provides an intraoperative display forsurgical systems to provide adaptation and adjustability or overlaidinstrument information. One aspect provides functional overlay ofinstrument critical operations or parameters to clearly represent asurgical stapler or energy device or aspects of each of its interactionwith tissue during a surgical procedure. Overlaid data may be adjustedby aspects detected by the surgical hub to modify the overlay from theinformation merely detected by the source instrument to add context. Thedisplays may be adjusted or modified by the user and as a result alsoresult in modifications of the instrument being monitored operation.

FIGS. 36-68 describe an intraoperative display system for use during asurgical procedure. The system comprises a surgical monitor with anintraoperative data display of a surgical area. An advanced energygenerator is coupled to an advanced energy surgical instrument. Theadvanced energy surgical instrument employs radio frequency (RF) energyand ultrasonic energy during a surgical procedure on a patient. Asurgical hub is coupled to the advanced energy generator and to thesurgical monitor. The surgical hub provides a live feed of the surgicalarea to the surgical monitor to display the live feed of the surgicalarea by the intraoperative data display. The intraoperative data displaydisplays a view of the surgical area including the advanced energysurgical instrument grasping tissue and a panel overlay that displaysinformation specific to the advanced energy surgical instrument.

One aspect, the intraoperative data display shows an end effector of thesurgical instrument grasping tissue and a panel overlay displaying caseinformation, systems notifications, or device panels, or any combinationthereof, overlaid over the live surgical feed. A location, opacity, sizeand placement of the panel overlay is customized. The panel overlay isconfigured to be turned on or off individually or turned on/off as agroup. The panel overlay is further configured to change dynamically toshow state changes such as device activation or power level adjustment.The panel overlay depicts optimal device performance (ODP) guide imagesor other instructions for use (IFU)/informational sources.

In various aspects the panel overlay comprises at least one of datainput information from capital equipment, generators, insufflator, smokeevacuator, electronic health record, laparoscope, computer, surgicaldevices, wired and wirelessly connected devices, surgeon profilepreferences that may be saved, recalled or edited, or any combinationthereof. The panel overlay may include case information including atleast one of Patient Name, Surgeon Name, Case Time, or InstrumentActivations, or combinations thereof. The panel overlay may includesystem notifications including at least one of connect instrumentstatus, minor error alert, medium error alert, or major error alert, orany combination thereof. The panel overlay may include informationassociated with the surgical instrument connected to the system toprovide advanced hemostasis. The panel overlay may include a visiblepatient panel overlay. The panel overlay may include a device paneloverlay comprising at least one of device name, device settings, ordevice supplemental features, or any combination thereof. The paneloverlay may include a plurality of panel overlays in a stackedconfiguration. The panel overlay may include a plurality of paneloverlays in an expanded configuration. The panel overlay may displaydevice troubleshooting information. The panel overlay may display atleast one of alerts, warnings, device information, or device features,or any combination thereof.

In another aspect, the intraoperative data display comprises a secondaryconfigurable panel. The secondary configurable panel changes dynamicallybased on the selected customized laparoscopic overlay fields displayedin the surgical field of view of a live surgical feed area of theintraoperative data display. The customized laparoscopic overlay fieldscomprise at least one of a bottom edge panel, a top left corner panel, atop center panel, or a side edge panel, or any combination thereof.

FIG. 36 is a system diagram 3000 of a surgical suite comprising asurgical monitor with intraoperative data display 3002 of a surgicalarea. An advanced energy generator 3004 is coupled to a surgical hub3006 and to an advanced energy surgical instrument 3008. The advancedenergy surgical instrument 3008 employs RF energy and ultrasonic energyduring a surgical procedure on a patient 3010. The surgical hub 3006provides a live feed 3014 of the surgical area, which is displayed bythe intraoperative data display 3002. The intraoperative data display3002 displays a view of the surgical area including the advanced energysurgical instrument 3008 grasping tissue and a panel overlay 3012 thatdisplays information specific to the advanced energy surgical instrument3008.

FIG. 37 is an augmented image 3100 of a live feed of a surgical area asvisualized through a laparoscopic camera during a minimally invasivesurgical procedure displayed on an intraoperative data display 3002. Thelaparoscopic field of view shows an end effector 3108 of a surgicalinstrument 3008 grasping tissue 3110. The augmented image 3100 showsthree panel overlays displaying case information 3102, systemsnotifications 3104, and device panels 3106 overlaid over the livesurgical feed.

The panel overlays 3102, 3104, 3106 are displayed on the live surgicalfeed. The location, opacity, size and placement of the panel overlays3102, 3104, 3106 can be customized. The panel overlays 3102, 3104, 3106may be turned on or off individually or turned on/off as a group. Thepanel overlays 3102, 3104, 3106 may be opaque or have varying levels oftransparency. The panel overlays 3102, 3104, 3106 may include data inputinformation from capital equipment, generators, insufflator, smokeevacuator, electronic health record, laparoscope, computer, surgicaldevices, wired and wirelessly connected devices. Surgeon profilepreferences may be saved, recalled or edited. In other aspects, generalscreen settings including overall screen settings and fonts for paneloverlay 3102, 3104, 3106 sizing may be configurable based upon surgeonpreferences via a staff console. The panel overlays 3102, 3104, 3106features on all screen displays may be enables/disabled, or bypassedthrough a dedicated physical switch, for example.

FIG. 38 is a detailed view of the case information panel overlay 3102shown in FIG. 37. The case information panel overlay 3102 may beselectively enabled/disabled via the staff console. Specific details ofthe case information panel overlay 3102 including Patient Name, SurgeonName, Case Time, Instrument Activations, etc., may be selectivelyenabled/disabled based on the staff console settings Other aspects ofthe case information panel overlay 3102 may include, for example,enabling the screen at the start of the surgery and disabling the screenafter a specific criteria is met (surgery time elapsed number of deviceactivations, etc.).

FIG. 39 is a detailed view of the systems notifications panel overlay3104 shown in FIG. 37. The systems notifications panel display overlay3104 may comprise a warning symbol that may be variable based on theseverity of the warning/alert/information. The message may bepotentially based on any combination of: connected capital equipment,connected devices (medical, wired, wireless, etc.), or surgical hubdevice (this device). The notification disappears based on notificationdwell time or alert status resolution.

FIG. 40 is an image of several examples of systems notifications paneloverlays 3104. The systems notifications panel overlays 3104 may includeconnect instrument 3112, minor error alert 3114, medium error alert3116, and major error alert 3118, for example.

FIG. 41 is a detailed view of the device panels overlay 3106 shown inFIG. 37. The example device panels overlay 3106 shown in FIG. 41,displays information associated with a combination ultrasonic/RFinstrument connected to the system 3000 (FIG. 36) to provide advancedhemostasis. The device panels overlay 3106 provide a Thermal indicationand a Ready indication, for example.

In various aspects, the device panels overlay 3106 provides a visualconcept influenced by the Ottava framework and may be selectivelyenabled/disabled via the staff console. The staff console also canselectively enable/disable individual panels such as, for example,energy and surgical stapler. In one aspect, the device panels overlay3106 only appears if a relevant instrument is connected to the system3000 (FIG. 36). The device panels overlay 3106 are not visible ifnothing is connected to the system 3000 (FIG. 36). The space occupied bythe device panels overlay 3106 is configurable (e.g., left, center,right aligned). The device panels overlay 3106 also may be configured todisplay connectivity to handheld instruments. In other aspects, thedevice panels overlay 3106 may be configurable in the space along thetop/bottom of the display. The device panels overlay 3106 may displaymultiple instrument devices connected simultaneously. In one aspect, thedevice panels overlay 3106 may display information only for deviceswhich are actively being used by the surgeon such that connected, butinactive device displays are hidden.

In one aspect, the size of the device panels overlay 3106 may beconfigurable based on the features that are enabled, such as the Thermalexample described herein. These features may be enabled or disabledbased upon intrinsic device design (e.g., future models), as well asenabled/disabled by the surgeon, or paid subscription.

FIG. 42 is an augmented image 3120 of a live feed of a surgical area asvisualized through a laparoscopic camera during a minimally invasivesurgical procedure displayed on an intraoperative data display 3122.showing an end effector 3108 of a surgical instrument 3008 graspingtissue 3110 and a screen with a visible patient panel overlay 3124, inaccordance with at least one aspect of the present disclosure. Thevisible patient panel overlay 3124 may require an additional applicationto display content. The intraoperative data display 3122 also displaysthe case information panel overlay 3102 and the device panels overlay3106. In the illustrated example, the systems notifications paneloverlay 3104 is hidden.

FIG. 43 is a schematic view of an energy device image panel 3106architecture. In one aspect, the energy device image panel 3106 includesa device name 3132, device settings 3134, and device supplementalfeatures 3136, for example.

FIG. 44 is an image 3140 of supplemental devicealerts/warning/information in a stacked configuration. The supplementaldevice alerts/warnings/information display may be stacked. Thesupplemental device alerts/warnings/information display also may bestacked vertically to cover the main device panel, or may be expandedhorizontally. As shown in the example of FIG. 44, the energy devicepanel overlay 3106 is stacked vertically below a system ready 3142display.

FIG. 45 is an image 3150 of supplemental devicealerts/warning/information in an expanded configuration. Thesupplemental device alerts/warnings/information display may be expanded.The expanded display 3144 may be expanded vertically or horizontally. Asshown in the example of FIG. 45, the expanded display 3144 providesadditional information about the instrument device. As shown in theexample is FIG. 45, the expanded display 3144 shows an image of asurgical instrument and the instructions to activate the instrument for2 seconds t run test.

FIG. 46 is an instrument state image panel 3160 showing how instrumentpanel states change dynamically to show state changes such as deviceactivation or power level adjustment. The left side 3162 of theinstrument state image panel 3160 shows the default state of theinstrument. The right side 3164 of the instrument state image panel 3160shows the active state of the instrument. In the example shown in FIG.46, a first panel 3166 on the right side 3164 of the instrument stateimage panel 3160 shows a dynamic change in the instrument activation anda second panel 3168 of the right side 3164 on the instrument state imagepanel 3160 shows a dynamic change in the instrument power level.

FIG. 47 is a system diagram 3170 of translating generator 3066 alertsand warnings 3174 to a laparoscopic monitor 3172 and displayed on alocal interface.

FIG. 48 is a diagram 3180 of a series of screens 3182 of existing alertsshown on a current generator that are transmitted to a surgical hub,which then displays them as a series of screens 3184 on a localinterface.

FIG. 49 is a schematic diagram of a system 3200 comprising a generator3202 in communication with a digital hub 3204, which then displaysscreen data and alert data 3220 on a local interface 3206 such as alaparoscopic screen 3216. The generator 3202 comprises a controller 3208and a screen 3210. The controller 3208 of the generator 3202 transmitsdata 3218 to a controller 3212 of the digital hub 3204. The digital hub3204 also comprises a screen 3214. The controller 3212 of the digitalhub 3204 transmits screen data and alert data 3220 to the laparoscopicscreen 3216 of the local interface 3206.

FIG. 50 is an augmented image 3230 of a live feed of a surgical area asvisualized through a laparoscopic camera during a minimally invasivesurgical procedure displayed on an intraoperative data display 3232. Thelaparoscopic field of view of the surgical area shows an overlay panel3234 depicting optimal device performance (ODP) guide images or otherinstructions for use (IFU)/informational sources. The augmented image3230 shows an on-screen display overlay panel 3234 shows a surgicaldevice 3250 comprising a grip housing 3244, trigger 3242, shaft 3238 andblade with tissue pad 3236. The shaft 3238 is rotated by a rotation nob3248. The surgical device 3250 also includes an energy button 3240 andan energy button 3246 with advanced hemostasis.

The overlay panel 3234 displays an image of a surgical device 3250 andODP guide images or other IFU/informational sources. The images may bepresented to the surgeon with surgical device information. The imagesmay be static or animated images. The images may provide generalsurgical device 3250 information (as shown), or context specificsurgical device information. For example, the bailout door of a surgicaldevice 3250 may include a sensor to detect removal. When the bailoutdoor is removed, the on-screen display overlay panel 3234 shows an imagewhich provides instructions on the proper usage of the bailoutmechanism. By way of another example, the surgeon encounters an alertwhile using the ultrasonic energy mode related to a surgical devicetechnique. The on-screen display overlay panel 3234 shows informationspecific to how to best use the surgical device 3250 to avoid thatalert.

FIG. 51 is a display screen 3300 showing an intraoperative data display3302 comprising a secondary bottom edge configurable panel 3312. Theintraoperative data display 3302 includes a surgical field of view of alive surgical feed 3303 with customized laparoscopic overlay fields3318, 3320, 3322 and a secondary bottom edge configurable panel 3312.The intraoperative data display 3302 displays a navigation button 3304to return back to the home screen. The intraoperative data display 3302also provides a selectable surgeon profile 3306. The surgeon profilepreferences may be saved, recalled, or edited. The intraoperative datadisplay 3302 also provides several connectivity status indicators 3308.Overlays associated with the intraoperative data display overlay fields3318, 3320, 3322 may be toggled with a display overlay toggle button3310.

The intraoperative data display 3302 also includes a secondaryconfigurable panel that changes dynamically based on the selectedcustomized laparoscopic overlay fields 3318, 3320, 3322 displayed in thesurgical field of view of a live surgical feed 3303 area of theintraoperative data display 3302 when the display overlay toggle button3310 is toggled in the ON position. In the example illustrated in FIG.51, a bottom edge configurable panel 3312 is displayed by selecting thebottom edge selection field 3322 in the surgical field of view of thelive surgical feed 3303 portion of the intraoperative data display 3302.Different portions of the display may be selected including, forexample, a top left corner selection field 3318, a top center selectionfield 3320, and a bottom edge selection field 3322.

The bottom edge configurable panel 3312 includes a panel alignment bar3314 to align the bottom edge configurable panel 3312 in the left,center, or right position, here shown in the left position. Helpindicators 3316 provide contextual information to the associated togglebutton 3324, 3326, 3328, 3330. The bottom edge selection field 3322 maybe configured using the configurable panel alignment button 3314, whichshifts the bottom edge alignment left, center, and right. In otheraspects, the bottom edge selection field 3322 may be moved top andbottom, for example. In addition to the configurable panel alignmentbutton 3314, the bottom edge configurable panel 3312 comprises a firsttoggle button 3324, which enables/disables the energy panel display forultrasonic/RF energy tool devices, including alerts, here shown in theON position. A second toggle button 3326 enables/disables display ofalerts only for ultrasonic/RF energy tool devices, here shown in the OFFposition. A third toggle button 3328 enables/disables display ofsurgical stapler tool devices, including alerts, here shown in the OFFposition. A fourth toggle button 3330 enables/disables display alertsonly for surgical stapler tool devices, here shown in the ON position.

FIG. 52 is an alternative bottom edge configurable panel 3400. Thealternative bottom edge configurable panel 3400 is displayed byselecting the bottom edge selection field 3282 in the surgical field ofview of a live surgical feed 3263 portion of the intraoperative datadisplay 3262 in FIG. 51. The alternative bottom edge configurable panel3400 may be configured for customization of energy status and alerts. Inthe illustrated example, the alternative bottom edge configurable panel3400 is displayed by toggling the display on monitor toggle button 3402in the ON position. A first toggle button 3404 enables/disables energydevice panel status, here shown in the ON position. A second togglebutton 3406 enables/disables energy device panel alerts, here shown inthe OFF position. A third toggle button 3408 enables/disables surgicalstapler device status, here shown in the OFF position. A fourth togglebutton 3410 enables/disables surgical stapler alerts, here shown in theOFF position. The alternative bottom edge configurable panel 3400includes a panel alignment bar 3412 to align the alternative bottom edgeconfigurable panel 3400 in the left, center, or right position, hereshown in the center position. FIGS. 53-55 described hereinbelow describeadditional configurable panels displayed by selecting other selectionfields in the surgical field of view of a live surgical feed portion ofthe intraoperative data display.

FIG. 53 is a display screen 3500 showing an intraoperative data display3502 comprising a secondary top left corner configurable panel 3522. Theintraoperative data display 3502 includes a surgical field of view of alive surgical feed 3503 with customized laparoscopic overlay fields3318, 3320, 3322 and a secondary top left corner configurable panel3522. The intraoperative data display 3502 includes a navigation buttonto return back to the home screen, a selectable surgeon profile, andconnectivity status indicators as described in connection with FIG. 51.Overlays associated with the intraoperative data display overlay fields3318, 3320, 3322 may be toggled with a display overlay toggle button3310.

The intraoperative data display 3502 also includes a secondaryconfigurable panel that changes dynamically based on the selectedcustomized laparoscopic overlay fields 3318, 3320, 3322 displayed in thesurgical field of view of a live surgical feed 3503 area of theintraoperative data display 3502 when the display overlay toggle button3310 is toggled in the ON position. In the example illustrated in FIG.53, a top left corner configurable panel 3522 is displayed by selectingthe top left corner selection field 3318 in the surgical field of viewof the live surgical feed 3503 portion of the intraoperative datadisplay 3502.

Selecting the top left corner selection field 3318 dynamically changesthe visual display. For example, selecting the top left corner selectionfield 3318,displays the case information overlay 3504, which, in thisexample, is the case information panel overlay screen 3102 shown inFIGS. 37 and 38. Various toggle buttons are shown alongside the rightvertical portion of the secondary top left corner configurable panel3522. As previously discussed, the display overlay toggle button 3310enables/disables the display of the overlay screens such as the caseinformation overlay 3504. A first toggle button 3506 enables/disablesall case information from the case information overlay 3504, here shownin the ON position. A second toggle button 3508 enables/disables patientname information from the case information overlay 3504, here shown inthe ON position. A third toggle button 3510 enables/disables surgeonname information from the case information overlay 3504, here shown inthe ON position. A fourth toggle button 3512 enables/disables case dateinformation from the case information overlay 3504, here shown in the ONposition. A fifth toggle button 3514 enables/disables total time for thecase information from the case information overlay 3504, here shown inthe ON position. A sixth toggle button 3516 enables/disables casemetrics, which may include total number of device activations, uses, orother device metrics, from the case information overlay 3504, here shownin the ON position.

FIG. 54 is a display screen 3600 showing an intraoperative data display3602 comprising a secondary top center configurable panel 3622. Theintraoperative data display 3602 includes a surgical field of view of alive surgical feed 3603 with customized laparoscopic overlay fields3318, 3320, 3322 and a secondary top center configurable panel 3622. Theintraoperative data display 3602 includes a navigation button to returnback to the home screen, a selectable surgeon profile, and connectivitystatus indicators as described in connection with FIG. 51. Overlaysassociated with the intraoperative data display overlay fields 3318,3320, 3322 may be toggled with a display overlay toggle button 3310.

The intraoperative data display 3602 also includes a secondaryconfigurable panel that changes dynamically based on the selectedcustomized laparoscopic overlay fields 3318, 3320, 3322 displayed in thesurgical field of view of a live surgical feed 3603 area of theintraoperative data display 3602 when the display overlay toggle button3310 is toggled in the ON position. In the example illustrated in FIG.54, a top center configurable panel 3622 is displayed by selecting thetop center selection field 3320 in the surgical field of view of thelive surgical feed 3603 portion of the intraoperative data display 3602.

Selecting the top center selection field 3320 dynamically changes thevisual display. For example, selecting the top center selection field3320, displays the systems notifications panel overlay 3604, which, inthis example, is the systems notifications panel overlay 3104 shown inFIG. 40. Various toggle buttons are shown alongside the right verticalportion of the secondary top center configurable panel 3622. Aspreviously discussed, the display overlay toggle button 3310enables/disables the display of the overlay screens such as the systemsnotifications panel overlay 3604. A first toggle button 3506enables/disables all alerts for the systems notifications panel overlay3604, here shown in the ON position. A second toggle button 3608enables/disables informational alerts for the informational notificationbar 3616 of the systems notifications panel overlay 3604, here shown inthe ON position. A third toggle button 3610 enables/disables minoralerts for the minor error alert bar 3618 of the systems notificationspanel overlay 3604, here shown in the ON position. A fourth togglebutton 3612 enables/disables medium alerts for the medium error alertbar 3620 of the systems notifications panel overlay 3604, here shown inthe ON position. A fifth toggle button 3614 enables/disables the majoralerts for the major error alert bar 3622 of the systems notificationspanel overlay 3604, here shown in the ON position. Dynamic shading ofthe toggle button, e.g., the fifth toggle button 3614 as shown, providesa visual cue based on the selection status.

FIG. 55 is display screen 3700 showing an intraoperative data display3702 comprising a secondary side edge configurable panel 3722. Theintraoperative data display 3702 includes a surgical field of view of alive surgical feed 3703 with customized laparoscopic overlay fields3318, 3320, 3322, 3716 and a secondary side edge configurable panel3722. The intraoperative data display 3702 includes a navigation buttonto return back to the home screen, a selectable surgeon profile, andconnectivity status indicators as described in connection with FIG. 51.Overlays associated with the intraoperative data display overlay fields3318, 3320, 3322, 3716 may be toggled with a display overlay togglebutton 3310.

The intraoperative data display 3702 also includes a secondaryconfigurable panel that changes dynamically based on the selectedcustomized laparoscopic overlay fields 3318, 3320, 3322, 3716 displayedin the surgical field of view of a live surgical feed 3703 area of theintraoperative data display 3702 when the display overlay toggle button3310 is toggled in the ON position. In the example illustrated in FIG.55, a side edge configurable panel 3722 is displayed by selecting theside edge selection field 3716 in the surgical field of view of the livesurgical feed 3703 portion of the intraoperative data display 3702.

Selecting the side edge selection field 3716 dynamically changes thevisual display. For example, selecting the side edge selection field3716, displays the visible patient panel overlay 3704, which, in thisexample, is the visible patient panel overlay 3124 shown in FIG. 42.Various toggle buttons are shown alongside the right vertical portion ofthe secondary side edge configurable panel 3722. As previouslydiscussed, the display overlay toggle button 3310 enables/disables thedisplay of the overlay screens such as the visible patient panel overlay3704. A first toggle button 3706 enables/disables all alerts for thevisible patient panel overlay 3704 information, here shown in the ONposition. A second toggle button 3708 enables/disables gas informationfor the visible patient panel overlay 3704, here shown in the ONposition. A third toggle button 3710 enables/disables visible patientinformation for the visible patient panel overlay 3704, here shown inthe ON position. A fourth toggle button 3712 enables/disablesinformation from another module (Module 3) for the visible patient paneloverlay 3704, here shown in the ON position. A fifth toggle button 3714enables/disables information from yet another module (Module 4) for thevisible patient panel overlay 3704, shown in the ON position.

FIG. 56 is a series of image panels 3800 displaying devicetroubleshooting information. A first image panel 3802 displaysinstructions to test the instrument. In the illustrated example, theinstruction to the instrument is “Activate instrument for 2 seconds torun test.” A second image panel 3804 indicates that testing is inprogress and a third image panels 3806 shows the testing status. Afourth image panel 3808 provides instructions for troubleshooting withstatic images, animations, GIFS, or videos. In the illustrated example,the image panel 3808 displays an animation to “Tighten Assembly.” Afifth image panel 3810 indicates an action for the instrument under testand, in the illustrated example, informs to “Open jaws during test.”

FIG. 57 is a series of image panels 3900 displaying articulatingsurgical stapler features. A first image panel 3902 indicates surgicalstapler articulation angle of the surgical stapler device. The angle maybe shown numerically or visually. Second and third image panels 3904,3906 indicate articulation angle updates based on device status and alsoshow articulation direction. In one aspect, each time the articulationbutton is pressed or articulation angle changes, an updated measurementis sent to the digital hub 3204 (FIG. 49), 3006 (FIG. 36). The digitalhub 3204 (FIG. 49), 3006 (FIG. 36) then uses the updated measurement toupdate the reflected angle as shown on the image panels 3902, 3904,3906.

A fourth image panel 3908 is a standardized countdown indicator thatappears when the surgical stapler jaw is closed. The countdowndynamically changes based on time. The device may be fired any timeduring the countdown sequence. A fifth image panel 3910 indicates thatthe device is ready to fire. Sixth, seventh, and eight image panels3912, 3914, 3916 indicate the knife position along the sled. The knifeposition is shown in gray over the an illustration of a cartridge 3918and dynamically changes based on the device. An illustration of acartridge 3918 may be generic or specific to the cartridge installed inthe surgical stapler. The knife position/surgical stapler image panels3912, 3914, 3916 may dynamically change based on the type and size ofthe surgical stapler wirelessly connected. A knife position algorithmexecutes after or as the firing trigger of the surgical stapler isdepressed and as the surgical stapler begins firing, the knife and sledbegin to travel down the length of the surgical stapler A ninth imagepanel 3920 indicates that the operation is complete.

Each of the supplemental features displayed by the corresponding imagepanels 3902-3920 of the connected device dynamically update based on thecurrent status of the device.

FIG. 58 is an alert/warning/information image panel 3922 displaying thatthe articulation limit has been reached.

FIG. 59 is an alert/warning/information image panel 3924 displaying thatthe device is in lockout mode.

FIG. 60 is an alert/warning/information image panel 3926 displaying thatthe device cannot articulate when jaws are closed.

FIG. 61 is a device image panel 4000 showing articulating surgicalstapler features. The device image panel 4000 includes a device panelname 4002, here shown as surgical stapler 3000, and a device panelsupplemental feature 4004, here shown as the countdown indicator.

FIG. 62 is a stacked alert/warning/information image panel 4010displayed in a stacked configuration with device alert displaying thatthe articulation limit has been reached. The stacked image panel 4010includes a top image panel 4012 indicating that the articulation limithas been reached. The stacked image panel 4010 includes a bottom imagepanel 4014 that indicates the name and type of the device, articulationangle, and articulation direction.

FIG. 63 is a schematic diagram of a system 4020 comprising a surgicalstapler in communication with a digital hub over Bluetooth to execute analgorithm for the countdown timer image panel 4000 shown in FIGS. 57 and61. The surgical stapler 4022 is in communication with the digital hub4024 over a wireless network, such as Bluetooth, for example. Thesurgical stapler a controller 4028 and the digital hub 4024 comprises acontroller 4030. The surgical stapler controller 4028 transmits wirelessdata 4026 to the digital hub controller 4030. When the surgical stapler4022 clamps and sends a clamp status message to the digital hub 4024,the digital hub 4024 uses the clamp status message as a start command toinitiate a countdown timer. If an unclamp message is received, the timerrestarts to 0 and stop incrementing. If the timer reaches a set time,the timer is stopped and a flag is set.

FIG. 64 is a series of device image panels/alerts 4030 displayingultrasonic instrument features. The device image panels are expandableor collapsible depending upon the features that are enabled or disabledby the user and/or the features that are enabled or disabled by thedevice connected. By way of example, a first device image panel 4032 canbe expanded into the second device image panel 4034, which displays adefault thermal image panel 4036 and a high thermal image panel 4038.The high thermal image panel 4038 displays the status of the devicetemperature by direct means (temperature sensing) or indirectly viaother methods (algorithm based on time or other means). With referenceto the default thermal image panel 4036, the thermal algorithm may bebased on device activation status and total time elapsed, as well astime elapsed since the last activation. In other implementations, thethermal algorithm may be based on device characteristics and/or statussuch as resonant frequency, inferred state of the device, or directmeasurement. Thermal status is displayed only while the device is notcurrently activating. While the device is inactive, the display isgrayed out. The thermal status is displayed at all times in the highthermal image panel 4038 and corresponds to the current state of thedevice. In the illustrated example, the device is an ultrasonicinstrument.

A third device image panel 4040 changes color as the device is beingused to illustrate the mode of device operation as shown by internalimage panels 4042, 4044, 4046. A fourth device image panel 4048 displaysinstrument alerts and associated images and text for that alert as shownin the alert image panel 4050. The alert image panel 4050 may providevisual indicator for the alert. Alerts may be composed of only text,only images, or a combination of text and images.

FIG. 65 is a chart 4060 describing pairing a surgical staplerinstrument. The surgical stapler pairing may be implemented as userselectable pairing, connect on powerup, connect on powerup with buttonpress, or RFID token. For user selectable pairing, the user would starta pairing mode on the capital equipment via a touchscreen. The capitalequipment would then attempt to identify all valid devices of that typewithin a region, and the user would simply select the devices that theywould like to connect to. Broadcasting would continue even after timehas elapsed. Connect on powerup pairing may enable Bluetooth radio onpowerup, and may be turned off if it is not connected to the clientwithin a predetermined time. The capital equipment constantly scans fornew devices and connects to them. Connect on powerup with button pressrequires the Bluetooth radio to be enabled on powerup and if it is notconnected to the client within a predetermined time, turn the radio off.The capital equipment has a pairing button that once pressedscans/connects to the device. In RFID token pairing, the user utilizesan RFID card, packaged with each device, as a method for scanning in thenew device and pairing with the capital equipment.

FIG. 66 is an image of a screen 4070 displaying pairing devicesinformation. The screen 4070 provides system wide ability toconnect/disconnect a variety of wired and wireless devices/equipment.

FIG. 67 is an image of a wireless surgical device 4080 comprising aunique identifier for pairing wireless devices.

FIG. 68 is an image of a screen 3090 displaying a link to optimal deviceperformance (ODP) guide images or other electronic instructions for use(e-IFU). The user manual 4092 may link to e-IFU or ODP.

With reference also to FIGS. 1-68, FIG. 69 is a diagram of an augmentedreality method 5000 employing a surgical instrument 77 and an augmentedreality display 89 for use during a surgical procedure, according to oneaspect of this disclosure. The method 5000 may be employed inconjunction with any of the augmented displays shown in FIGS. 11-48,50-62, 64-66, and 68 and may be implemented with any of the systemsshown in FIGS. 1-10, 49, and 63. In one aspect, the method 500 providesoverlay of operational aspects or functions of a surgical instrument 77onto the surgical laparoscopic video stream. The overlay may be relatedto the operation of one of the surgical instruments 77 being activelyvisualized and the overlay combines aspects of the tissue/organinteraction with functional data received from the surgical instrument77. In another aspect, the surgical instrument 77 may be grasper orclamp and the aspect of the tissue could be incomplete capture of thetissue along with the clamp status or magnitude of the clamp. In anotheraspect, the surgical instrument 77 may be a surgical stapler and theaspect of the tissue may be tissue capture location or tissuecompression and the aspect of the surgical stapler may be clamping orfiring sufficiency, or other parameters. In another aspect, the surgicalinstrument 77 may be an advanced energy device and the tissue parametermay be impedance, cautery status, bleeding magnitude and the function ofthe surgical instrument 77 could be energy level, timing, clamppressure, among other parameters.

In one aspect, the method 5000 is directed to overlay of data accordingto surgical instrument 77 utilization. According to the method 5000, animaging device 38 captures 5002 a real image of a surgical area during asurgical procedure. A processor 85 receives 5004 functional data fromthe surgical instrument 77, determines 5006 an overlay related to anoperational aspect of the surgical instrument 77, and combines 5008 anaspect of tissue in the surgical area with the functional data receivedfrom the surgical instrument 77. The augmented reality display 89, orlocal display 67, presents 5010 the overlay of the operational aspect ofthe surgical instrument 77 onto the real image of the surgical area. Thefunctional data for the surgical instrument 77 may be received from thesurgical instrument 77 directly or a surgical hub coupled processor orserver.

With reference also to FIGS. 1-68, FIG. 70 is a diagram of an augmentedreality method 5100 employing a surgical instrument 77 and an augmentedreality display 89 for use during a surgical procedure, according to oneaspect of this disclosure. The method 5100 may be employed inconjunction with any of the augmented displays shown in FIGS. 11-48,50-62, 64-66, and 68 and may be implemented with any of the systemsshown in FIGS. 1-10, 49, and 63.

In one aspect, the method 5100 is directed to overlay of data accordingto surgical instrument 77 utilization. The processor 85 monitors 5102the performance of the surgical instrument 77 during a surgicalprocedure. The processor 85 determines 5104 the usage of the surgicalinstrument 77. The augmented reality display 89 displays 5112 alertsrelative to the surgical instrument 77 performance considerations. Theprocessor 85 determines 5122 collateral events, displays 5132 a statusoverlay of the surgical instrument 77, and displays 5134 an indicationof the surgical instruments 77 on the augmented reality display 89.

Once the processor 85 determines 5104 the usage of the surgicalinstrument 77, the processor 85 determines 5106 whether the tissuegrasped in the jaws of the surgical instrument 77 is within a range ofthe jaws and determines 5108 whether the tissue is properly balancedwithin the jaws of the surgical instrument 77 and displays 5118incompatible usage alerts according to the state of the usage of thesurgical instrument 77. If the tissue is out of range, the processor 85displays 5116 a tissue out of range alert on the augmented realitydisplay 8. If the tissue is incorrectly balanced within the jaws of thesurgical instrument 77, the processor 85 displays 5118 an incorrectbalance alert on the augmented reality display 89. As part ofdetermining 5104 the usage of the surgical instrument 77, the processor85 determines if the tissue capture between the jaws of the surgicalinstrument 77 is complete and if not displays 5110 an alert ofincomplete tissue capture.

According to the method 5000, the processor 85 determines 5122collateral events such as tissue tension and foreign object detection.If the processor 85 determines 5124 that the tissue tension is too high,the augmented reality display 89 displays 5126 a tissue tension alert.If the processor 85 detects 5128 a foreign object in the jaws of thesurgical instrument 77, the augmented reality display 89 displays 5130 aforeign object detected alert. In any case, the augmented realitydisplay 89 displays 5132 the surgical instrument 77 status overlayaccording to the results of the above mentioned determinations of tissuetension and foreign object detection. Finally, the augmented realitydisplay 89 displays 5134 the surgical instrument 77 indication.

In accordance with either the method 5000, 5100 shown in FIGS. 69-70,the processor configures the data overlay to be displayed by theaugmented reality display 89 as shown in FIGS. 36-68 above. Also, theaugmented reality display 89 displays the dynamic status of the surgicalinstrument 77 function as shown in FIGS. 36-68.

FIG. 71 is an image of a staff view screen 5600 displaying customizedoverlays information. The staff view screen 5600 displays primarysurgical display 5602 that enables customization of surgical overlaysand includes three main screen portions. A first screen portion 5604displays a surgeon preset to enable entering surgeon presets or defaultsettings that can be saved, changed, or deleted. The first screenportion 5604 also includes a section that enables display overlays toturned On or Off and an overlay size portion that enables the adjustmentof the primary surgical display 5602 from smaller to larger in threesettings of Small, Medium, and Large.

A second screen portion 5606 of the primary surgical display 5602 isdisplay to the right of the first screen portion. The second screenportion 5606 displays case information and overall device use date andenables editing the display of the case information. A right chevron5612 can be tapped to access more granular ability to turn on/offindividual overlays. A virtual switch slider button 5612 is used to turnon/off a group of overlays.

A third screen portion 5608 of the primary surgical display 5602 isdisplayed below the second screen portion 5606. The third screen portion5608 displays energy panels and device alerts, and enables editing thedisplay of the device panels. Similar to the second screen portion 5606,the third screen portion 5608 includes a right chevron 5616 that can betapped to access more granular ability to turn on/off individualoverlays and a virtual switch slider button 5618 to turn on/off a groupof overlays.

A fourth screen portion 5610 of the primary surgical display 5602 isdisplayed below the third screen portion 5608. The fourth screen portion5610 displays all system notifications and enables editing of the systemnotifications. Similar to the second and third screen portions 5606,5608, the fourth screen portion 5610 includes a right chevron 5620 thatcan be tapped to access more granular ability to turn on/off individualoverlays and a virtual switch slider button 5622 to turn on/off a groupof overlays.

Tappable icons are provide at the bottom of the primary surgical display5602 to provide additional functionality. For example, one tappable icon5624 enables navigation to staff view screens.

FIG. 72 is an image of a staff view screen 5700 displaying detailedcustomization pop-up information. The staff view screen 5700 displays anedit case overlay screen 5702 over the primary surgical display 5602screen. The edit case information overlay screen 5702 allows the abilityto turn on/off individual overlay components. The edit case informationoverlay screen 5702 includes a series of virtual switch slider buttons5704 to enable editing the patient first and last name, the surgeonfirst and last name, the time and date of the case (e.g., surgicalprocedure), the case time, number of instrument activations, or staplefirings, for example.

FIG. 73 is an image of a staff view screen 5800 displaying staff viewtroubleshooting pop-up information. The staff view screen 5800 displaysa staff view troubleshooting pop-up screen 5802 to providetroubleshooting information about system devices and components. Atappable icon 5084 provides device connection status and indicates ifthe system is properly connected to other capital equipment or devices.The troubleshooting pop-up screen 5802 shows a check ultrasonicconnection generator error screen with a static image 5806 of anultrasonic generator 5808. Additional troubleshooting screens withstatic images, gifs or animations may be provided. Troubleshootingscreens may have multiple steps or multiple screens that a user canadvance (not illustrated).

FIG. 74 is an image of a primary surgical display interactions screen5900 displaying primary surgical display interactions. The primarysurgical display interactions screen 5900 shows an instruments panelbehavior screen 5902 with an overlay screen 5904 comprising threesections. A first section of the overlay screen 5904 shows deviceactivation information including an instrument activation default panel5906 and an instrument minimum active panel 5908. Information includespanel background changes from dark to light in activated state andactive mode shown in dark text and inactive mode shown in faded opacity,for example.

A second section of the overlay screen 5904 shows instrument disableactivation information including an instrument disable activation panel5910, an alarm status panel 5912, and a disabled instrument panel 5914.When certain alarms are triggered, the instrument panel is grayed out toindicate that activation is disabled. This may apply only when the useris locked out of the device due to alarm status.

A third section of the overlay screen 5904 shows minimize informationincluding a generic instrument default panel 5916 and a minimized panel5918. The panels are minimized to a predetermined size after apredetermined period. The instrument type remains on the panel and thepanel returns to default view when activation or notification occurs.

Situational awareness is the ability of some aspects of a surgicalsystem to determine or infer information related to a surgical procedurefrom data received from databases and/or instruments. The informationcan include the type of procedure being undertaken, the type of tissuebeing operated on, or the body cavity that is the subject of theprocedure. With the contextual information related to the surgicalprocedure, the surgical system can, for example, improve the manner inwhich it controls the modular devices (e.g., a robotic arm and/orrobotic surgical tool) that are connected to it and providecontextualized information or suggestions to the surgeon during thecourse of the surgical procedure.

FIG. 75 illustrates a timeline of a situational awareness surgicalprocedure. FIG. 75 illustrates a timeline 5200 of an illustrativesurgical procedure and the contextual information that a surgical hub5104 can derive from the data received from the data sources 5126 ateach step in the surgical procedure. The timeline 5200 depicts thetypical steps that would be taken by the nurses, surgeons, and othermedical personnel during the course of a lung segmentectomy procedure,beginning with setting up the operating theater and ending withtransferring the patient to a post-operative recovery room. Thesituationally aware surgical hub 5104 receives data from the datasources 5126 throughout the course of the surgical procedure, includingdata generated each time medical personnel utilize a modular device 5102that is paired with the surgical hub 5104. The surgical hub 5104 canreceive this data from the paired modular devices 5102 and other datasources 5126 and continually derive inferences (i.e., contextualinformation) about the ongoing procedure as new data is received, suchas which step of the procedure is being performed at any given time. Thesituational awareness system of the surgical hub 5104 is able to, forexample, record data pertaining to the procedure for generating reports,verify the steps being taken by the medical personnel, provide data orprompts (e.g., via a display screen) that may be pertinent for theparticular procedural step, adjust modular devices 5102 based on thecontext (e.g., activate monitors, adjust the FOV of the medical imagingdevice, or change the energy level of an ultrasonic surgical instrumentor RF electrosurgical instrument), and take any other such actiondescribed above.

First 5202, the hospital staff members retrieve the patient's EMR fromthe hospital's EMR database. Based on select patient data in the EMR,the surgical hub 5104 determines that the procedure to be performed is athoracic procedure.

Second 5204, the staff members scan the incoming medical supplies forthe procedure. The surgical hub 5104 cross-references the scannedsupplies with a list of supplies that are utilized in various types ofprocedures and confirms that the mix of supplies corresponds to athoracic procedure. Further, the surgical hub 5104 is also able todetermine that the procedure is not a wedge procedure (because theincoming supplies either lack certain supplies that are necessary for athoracic wedge procedure or do not otherwise correspond to a thoracicwedge procedure).

Third 5206, the medical personnel scan the patient band via a scanner5128 that is communicably connected to the surgical hub 5104. Thesurgical hub 5104 can then confirm the patient's identity based on thescanned data.

Fourth 5208, the medical staff turns on the auxiliary equipment. Theauxiliary equipment being utilized can vary according to the type ofsurgical procedure and the techniques to be used by the surgeon, but inthis illustrative case they include a smoke evacuator, insufflator, andmedical imaging device. When activated, the auxiliary equipment that aremodular devices 5102 can automatically pair with the surgical hub 5104that is located within a particular vicinity of the modular devices 5102as part of their initialization process. The surgical hub 5104 can thenderive contextual information about the surgical procedure by detectingthe types of modular devices 5102 that pair with it during thispre-operative or initialization phase. In this particular example, thesurgical hub 5104 determines that the surgical procedure is a VATSprocedure based on this particular combination of paired modular devices5102. Based on the combination of the data from the patient's EMR, thelist of medical supplies to be used in the procedure, and the type ofmodular devices 5102 that connect to the hub, the surgical hub 5104 cangenerally infer the specific procedure that the surgical team will beperforming. Once the surgical hub 5104 knows what specific procedure isbeing performed, the surgical hub 5104 can then retrieve the steps ofthat procedure from a memory or from the cloud and then cross-referencethe data it subsequently receives from the connected data sources 5126(e.g., modular devices 5102 and patient monitoring devices 5124) toinfer what step of the surgical procedure the surgical team isperforming.

Fifth 5210, the staff members attach the EKG electrodes and otherpatient monitoring devices 5124 to the patient. The EKG electrodes andother patient monitoring devices 5124 are able to pair with the surgicalhub 5104. As the surgical hub 5104 begins receiving data from thepatient monitoring devices 5124, the surgical hub 5104 thus confirmsthat the patient is in the operating theater.

Sixth 5212, the medical personnel induce anesthesia in the patient. Thesurgical hub 5104 can infer that the patient is under anesthesia basedon data from the modular devices 5102 and/or patient monitoring devices5124, including EKG data, blood pressure data, ventilator data, orcombinations. Upon completion of the sixth step 5212, the pre-operativeportion of the lung segmentectomy procedure is completed and theoperative portion begins.

Seventh 5214, the patient's lung that is being operated on is collapsed(while ventilation is switched to the contralateral lung). The surgicalhub 5104 can infer from the ventilator data that the patient's lung hasbeen collapsed. The surgical hub 5104 can infer that the operativeportion of the procedure has commenced as it can compare the detectionof the patient's lung collapsing to the expected steps of the procedure(which can be accessed or retrieved previously) and thereby determinethat collapsing the lung is the first operative step in this particularprocedure.

Eighth 5216, the medical imaging device 5108 (e.g., a scope) is insertedand video from the medical imaging device is initiated. The surgical hub5104 receives the medical imaging device data (i.e., still image data orlive streamed video in real time) through its connection to the medicalimaging device. Upon receipt of the medical imaging device data, thesurgical hub 5104 can determine that the laparoscopic portion of thesurgical procedure has commenced. Further, the surgical hub 5104 candetermine that the particular procedure being performed is asegmentectomy, as opposed to a lobectomy (note that a wedge procedurehas already been discounted by the surgical hub 5104 based on datareceived at the second step 5204 of the procedure). The data from themedical imaging device 124 (FIG. 2) can be utilized to determinecontextual information regarding the type of procedure being performedin a number of different ways, including by determining the angle atwhich the medical imaging device is oriented with respect to thevisualization of the patient's anatomy, monitoring the number or medicalimaging devices being utilized (i.e., that are activated and paired withthe surgical hub 5104), and monitoring the types of visualizationdevices utilized.

For example, one technique for performing a VATS lobectomy places thecamera in the lower anterior corner of the patient's chest cavity abovethe diaphragm, whereas one technique for performing a VATS segmentectomyplaces the camera in an anterior intercostal position relative to thesegmental fissure. Using pattern recognition or machine learningtechniques, for example, the situational awareness system can be trainedto recognize the positioning of the medical imaging device according tothe visualization of the patient's anatomy. As another example, onetechnique for performing a VATS lobectomy utilizes a single medicalimaging device, whereas another technique for performing a VATSsegmentectomy utilizes multiple cameras. As yet another example, onetechnique for performing a VATS segmentectomy utilizes an infrared lightsource (which can be communicably coupled to the surgical hub as part ofthe visualization system) to visualize the segmental fissure, which isnot utilized in a VATS lobectomy. By tracking any or all of this datafrom the medical imaging device 5108, the surgical hub 5104 can therebydetermine the specific type of surgical procedure being performed and/orthe technique being used for a particular type of surgical procedure.

Ninth 5218, the surgical team begins the dissection step of theprocedure. The surgical hub 5104 can infer that the surgeon is in theprocess of dissecting to mobilize the patient's lung because it receivesdata from the RF or ultrasonic generator indicating that an energyinstrument is being fired. The surgical hub 5104 can cross-reference thereceived data with the retrieved steps of the surgical procedure todetermine that an energy instrument being fired at this point in theprocess (i.e., after the completion of the previously discussed steps ofthe procedure) corresponds to the dissection step.

Tenth 5220, the surgical team proceeds to the ligation step of theprocedure. The surgical hub 5104 can infer that the surgeon is ligatingarteries and veins because it receives data from the surgical staplingand cutting instrument indicating that the instrument is being fired.Similarly to the prior step, the surgical hub 5104 can derive thisinference by cross-referencing the receipt of data from the surgicalstapling and cutting instrument with the retrieved steps in the process.

Eleventh 5222, the segmentectomy portion of the procedure is performed.The surgical hub 5104 infers that the surgeon is transecting theparenchyma based on data from the surgical instrument, including datafrom a staple cartridge. The cartridge data may correspond to size ortype of staple being fired by the instrument. The cartridge data canindicate the type of tissue being stapled and/or transected fordifferent types of staples utilized in different types of tissues. Thetype of staple being fired is utilized for parenchyma or other tissuetypes to allow the surgical hub 5104 to infer that the segmentectomyprocedure is being performed.

Twelfth 5224, the node dissection step is then performed. The surgicalhub 5104 can infer that the surgical team is dissecting the node andperforming a leak test based on data received from the generatorindicating that an RF or ultrasonic instrument is being fired. For thisparticular procedure, an RF or ultrasonic instrument being utilizedafter parenchyma was transected corresponds to the node dissection step,which allows the surgical hub 5104 to make this inference. It should benoted that surgeons regularly switch back and forth between surgicalstapling/cutting instruments and surgical energy (i.e., RF orultrasonic) instruments depending upon the particular step in theprocedure because different instruments are better adapted forparticular tasks. Therefore, the particular sequence in which thestapling/cutting instruments and surgical energy instruments are usedcan indicate what step of the procedure the surgeon is performing. Uponcompletion of the twelfth step 5224, the incisions and closed up and thepost-operative portion of the procedure begins.

Thirteenth 5226, the patient's anesthesia is reversed. The surgical hub5104 can infer that the patient is emerging from the anesthesia based onthe ventilator data (i.e., the patient's breathing rate beginsincreasing), for example.

Lastly, fourteenth 5228, the medical personnel remove the variouspatient monitoring devices 5124 from the patient. The surgical hub 5104can thus infer that the patient is being transferred to a recovery roomwhen the hub loses EKG, BP, and other data from the patient monitoringdevices 5124. The surgical hub 5104 can determine or infer when eachstep of a given surgical procedure is taking place according to datareceived from the various data sources 5126 that are communicablycoupled to the surgical hub 5104.

In addition to utilizing the patient data from EMR database(s) to inferthe type of surgical procedure that is to be performed, as illustratedin the first step 5202 of the timeline 5200 depicted in FIG. 75, thepatient data can also be utilized by a situationally aware surgical hub5104 to generate control adjustments for the paired modular devices5102.

Various additional aspects of the subject matter described herein areset out in the following numbered examples:

Example 1: An augmented reality display system for use during a surgicalprocedure, the augmented reality display system comprising: an imagingdevice to capture a real image of a surgical area during the surgicalprocedure; an augmented reality display to present an overlay of anoperational aspect of a surgical instrument being actively visualizedonto the real image of the surgical area, wherein the overlay combinesaspects of tissue interaction in the surgical area with functional datareceived from the surgical instrument; and a processor to: receive thefunctional data from the surgical instrument; determine the overlayrelated to the operational aspect of the surgical instrument; andcombine the aspect of the tissue in the surgical area with thefunctional data received from the surgical instrument.

Example 2: The augmented reality display system of Example 1, whereinthe surgical instrument comprises an end effector with jaws to capturetissue therebetween and wherein the operational aspect of the surgicalinstrument is clamping tissue between the jaws of the end effector.

Example 3: The augmented reality display system of Example 2, whereinthe aspect of the tissue is appropriate capture of the tissue betweenthe jaws of the end effector.

Example 4: The augmented reality display system of any one of Examples2-3, wherein the aspect of the tissue is incorrectly positioned tissuebetween the jaws of the end effector.

Example 5: The augmented reality display system of any one of Examples2-4, wherein the aspect of the tissue is insufficiently captured tissuebetween the jaws of the end effector.

Example 6: The augmented reality display system of any one of Examples2-5, wherein the aspect of the tissue is tension of the tissue betweenthe jaws of the end effector.

Example 7: The augmented reality display system of Example 1, whereinthe surgical instrument comprises an end effector with jaws to capturetissue therebetween and wherein the operational aspect of the surgicalinstrument is overheating.

Example 8: The augmented reality display system of Example 1, whereinthe surgical instrument comprises an end effector with jaws to capturetissue therebetween and wherein the operational aspect of the surgicalinstrument is jaw closure position.

Example 9: The augmented reality display system of Example 8, whereinthe aspect of the tissue is movement or flow of the tissue between thejaws.

Example 10: The augmented reality display system of Example 1, whereinthe surgical instrument comprises an end effector with jaws to capturetissue therebetween and wherein the operational aspect of the surgicalinstrument is clamping on metal or foreign object between the jaws.

Example 11: The augmented reality display system of Example 10, whereinthe aspect of the tissue is geometric relationship of transections tothe tissue and to other firings.

Example 12: The augmented reality display system of 12 any one ofExamples 10-11, wherein the operational aspect of the surgicalinstrument is anvil orientation.

Example 13: The augmented reality display system of any one of Examples10-12, wherein the aspect of the tissue is tissue thickness.

Example 14: The augmented reality display system of any one of Examples10-13, wherein the operational aspect of the surgical instrument isclamping status of the jaws.

Example 15: The augmented reality display system of Example 1, whereinthe surgical instrument comprises an end effector with jaws to capturetissue therebetween and wherein the operational aspect of the surgicalinstrument is magnitude of the clamping status of the jaws.

Example 16: The augmented reality display system of Example 15, whereinthe aspect of the tissue is tissue compression.

Example 17: The augmented reality display system of Example 1, whereinthe surgical instrument comprises an end effector with jaws to capturetissue therebetween and wherein the operational aspect of the surgicalinstrument is sufficiency of clamping of the jaws.

Example 18: The augmented reality display system of Example 1, whereinthe surgical instrument comprises an end effector with jaws to capturetissue therebetween and wherein the operational aspect of the surgicalinstrument is firing sufficiency.

Example 19: The augmented reality display system of Example 1, whereinthe surgical instrument is an energy device and a tissue parameter isany one of impedances, cautery status, bleeding or magnitude, andwherein a functional aspect of the energy device is any one of energylevel, timing, or clamp pressure.

Example 20: An augmented reality display system for use during asurgical procedure, the augmented reality display system comprising: animaging device to capture a real image of a surgical area during thesurgical procedure; an augmented reality display to present an overlayof an operational aspect of a surgical instrument being activelyvisualized onto the real image of the surgical area, wherein the overlaycombines aspects of tissue interaction in the surgical area withfunctional data from the surgical instrument; and a processor to:receive the functional data from the surgical instrument; determine theoverlay related to the operational aspect of the surgical instrument;combine the aspect of the tissue in the surgical area with thefunctional data received from the surgical instrument; and configure theoverlay.

Example 21: An augmented reality display system for use during asurgical procedure, the augmented reality display system comprising: animaging device to capture a real image of a surgical area during thesurgical procedure; an augmented reality display to present an overlayof an operational aspect of a surgical instrument being activelyvisualized onto the real image of the surgical area, wherein the overlaycombines aspects of tissue interaction in the surgical area withfunctional data from the surgical instrument; and a processor to:receive the functional data from the surgical instrument; determine theoverlay related to the operational aspect of the surgical instrument;combine the aspect of the tissue in the surgical area with thefunctional data received from the surgical instrument; and displaydynamic status of the surgical instrument.

While several forms have been illustrated and described, it is not theintention of Applicant to restrict or limit the scope of the appendedclaims to such detail. Numerous modifications, variations, changes,substitutions, combinations, and equivalents to those forms may beimplemented and will occur to those skilled in the art without departingfrom the scope of this disclosure. Moreover, the structure of eachelement associated with the described forms can be alternativelydescribed as a means for providing the function performed by theelement. Also, where materials are disclosed for certain components,other materials may be used. It is therefore to be understood that theforegoing description and the appended claims are intended to cover allsuch modifications, combinations, and variations as falling within thescope of the disclosed forms. The appended claims are intended to coverall such modifications, variations, changes, substitutions,modifications, and equivalents.

The foregoing detailed description has set forth various forms of thedevices and/or processes via the use of block diagrams, flowcharts,and/or examples. Insofar as such block diagrams, flowcharts, and/orexamples contain one or more functions and/or operations, it will beunderstood by those within the art that each function and/or operationwithin such block diagrams, flowcharts, and/or examples can beimplemented, individually and/or collectively, by a wide range ofhardware, software, firmware, or virtually any combination thereof.Those skilled in the art will recognize that some aspects of the formsdisclosed herein, in whole or in part, can be equivalently implementedin integrated circuits, as one or more computer programs running on oneor more computers (e.g., as one or more programs running on one or morecomputer systems), as one or more programs running on one or moreprocessors (e.g., as one or more programs running on one or moremicroprocessors), as firmware, or as virtually any combination thereof,and that designing the circuitry and/or writing the code for thesoftware and or firmware would be well within the skill of one of skillin the art in light of this disclosure. In addition, those skilled inthe art will appreciate that the mechanisms of the subject matterdescribed herein are capable of being distributed as one or more programproducts in a variety of forms, and that an illustrative form of thesubject matter described herein applies regardless of the particulartype of signal bearing medium used to actually carry out thedistribution.

Instructions used to program logic to perform various disclosed aspectscan be stored within a memory in the system, such as dynamic randomaccess memory (DRAM), cache, flash memory, or other storage.Furthermore, the instructions can be distributed via a network or by wayof other computer readable media. Thus a machine-readable medium mayinclude any mechanism for storing or transmitting information in a formreadable by a machine (e.g., a computer), but is not limited to, floppydiskettes, optical disks, compact disc, read-only memory (CD-ROMs), andmagneto-optical disks, read-only memory (ROMs), random access memory(RAM), erasable programmable read-only memory (EPROM), electricallyerasable programmable read-only memory (EEPROM), magnetic or opticalcards, flash memory, or a tangible, machine-readable storage used in thetransmission of information over the Internet via electrical, optical,acoustical or other forms of propagated signals (e.g., carrier waves,infrared signals, digital signals, etc.). Accordingly, thenon-transitory computer-readable medium includes any type of tangiblemachine-readable medium suitable for storing or transmitting electronicinstructions or information in a form readable by a machine (e.g., acomputer).

As used in any aspect herein, the term “control circuit” may refer to,for example, hardwired circuitry, programmable circuitry (e.g., acomputer processor including one or more individual instructionprocessing cores, processing unit, processor, microcontroller,microcontroller unit, controller, digital signal processor (DSP),programmable logic device (PLD), programmable logic array (PLA), orfield programmable gate array (FPGA)), state machine circuitry, firmwarethat stores instructions executed by programmable circuitry, and anycombination thereof. The control circuit may, collectively orindividually, be embodied as circuitry that forms part of a largersystem, for example, an integrated circuit (IC), an application-specificintegrated circuit (ASIC), a system on-chip (SoC), desktop computers,laptop computers, tablet computers, servers, smart phones, etc.Accordingly, as used herein “control circuit” includes, but is notlimited to, electrical circuitry having at least one discrete electricalcircuit, electrical circuitry having at least one integrated circuit,electrical circuitry having at least one application specific integratedcircuit, electrical circuitry forming a general purpose computing deviceconfigured by a computer program (e.g., a general purpose computerconfigured by a computer program which at least partially carries outprocesses and/or devices described herein, or a microprocessorconfigured by a computer program which at least partially carries outprocesses and/or devices described herein), electrical circuitry forminga memory device (e.g., forms of random access memory), and/or electricalcircuitry forming a communications device (e.g., a modem, communicationsswitch, or optical-electrical equipment). Those having skill in the artwill recognize that the subject matter described herein may beimplemented in an analog or digital fashion or some combination thereof.

As used in any aspect herein, the term “logic” may refer to an app,software, firmware and/or circuitry configured to perform any of theaforementioned operations. Software may be embodied as a softwarepackage, code, instructions, instruction sets and/or data recorded onnon-transitory computer readable storage medium. Firmware may beembodied as code, instructions or instruction sets and/or data that arehard-coded (e.g., nonvolatile) in memory devices.

As used in any aspect herein, the terms “component,” “system,” “module”and the like can refer to a control circuit, computer-related entity,either hardware, a combination of hardware and software, software, orsoftware in execution.

As used in any aspect herein, an “algorithm” refers to a self-consistentsequence of steps leading to a desired result, where a “step” refers toa manipulation of physical quantities and/or logic states which may,though need not necessarily, take the form of electrical or magneticsignals capable of being stored, transferred, combined, compared, andotherwise manipulated. It is common usage to refer to these signals asbits, values, elements, symbols, characters, terms, numbers, or thelike. These and similar terms may be associated with the appropriatephysical quantities and are merely convenient labels applied to thesequantities and/or states.

A network may include a packet switched network. The communicationdevices may be capable of communicating with each other using a selectedpacket switched network communications protocol. One examplecommunications protocol may include an Ethernet communications protocolwhich may be capable permitting communication using a TransmissionControl Protocol/Internet Protocol (TCP/IP). The Ethernet protocol maycomply or be compatible with the Ethernet standard published by theInstitute of Electrical and Electronics Engineers (IEEE) titled “IEEE802.3 Standard”, published in December, 2008 and/or later versions ofthis standard. Alternatively or additionally, the communication devicesmay be capable of communicating with each other using an X.25communications protocol. The X.25 communications protocol may comply orbe compatible with a standard promulgated by the InternationalTelecommunication Union-Telecommunication Standardization Sector(ITU-T). Alternatively or additionally, the communication devices may becapable of communicating with each other using a frame relaycommunications protocol. The frame relay communications protocol maycomply or be compatible with a standard promulgated by ConsultativeCommittee for International Telegraph and Telephone (CCITT) and/or theAmerican National Standards Institute (ANSI). Alternatively oradditionally, the transceivers may be capable of communicating with eachother using an Asynchronous Transfer Mode (ATM) communications protocol.The ATM communications protocol may comply or be compatible with an ATMstandard published by the ATM Forum titled “ATM-MPLS NetworkInterworking 2.0” published August 2001, and/or later versions of thisstandard. Of course, different and/or after-developedconnection-oriented network communication protocols are equallycontemplated herein.

Unless specifically stated otherwise as apparent from the foregoingdisclosure, it is appreciated that, throughout the foregoing disclosure,discussions using terms such as “processing,” “computing,”“calculating,” “determining,” “displaying,” or the like, refer to theaction and processes of a computer system, or similar electroniccomputing device, that manipulates and transforms data represented asphysical (electronic) quantities within the computer system's registersand memories into other data similarly represented as physicalquantities within the computer system memories or registers or othersuch information storage, transmission or display devices.

One or more components may be referred to herein as “configured to,”“configurable to,” “operable/operative to,” “adapted/adaptable,” “ableto,” “conformable/conformed to,” etc. Those skilled in the art willrecognize that “configured to” can generally encompass active-statecomponents and/or inactive-state components and/or standby-statecomponents, unless context requires otherwise.

The terms “proximal” and “distal” are used herein with reference to aclinician manipulating the handle portion of the surgical instrument.The term “proximal” refers to the portion closest to the clinician andthe term “distal” refers to the portion located away from the clinician.It will be further appreciated that, for convenience and clarity,spatial terms such as “vertical”, “horizontal”, “up”, and “down” may beused herein with respect to the drawings. However, surgical instrumentsare used in many orientations and positions, and these terms are notintended to be limiting and/or absolute.

Those skilled in the art will recognize that, in general, terms usedherein, and especially in the appended claims (e.g., bodies of theappended claims) are generally intended as “open” terms (e.g., the term“including” should be interpreted as “including but not limited to,” theterm “having” should be interpreted as “having at least,” the term“includes” should be interpreted as “includes but is not limited to,”etc.). It will be further understood by those within the art that if aspecific number of an introduced claim recitation is intended, such anintent will be explicitly recited in the claim, and in the absence ofsuch recitation no such intent is present. For example, as an aid tounderstanding, the following appended claims may contain usage of theintroductory phrases “at least one” and “one or more” to introduce claimrecitations. However, the use of such phrases should not be construed toimply that the introduction of a claim recitation by the indefinitearticles “a” or “an” limits any particular claim containing suchintroduced claim recitation to claims containing only one suchrecitation, even when the same claim includes the introductory phrases“one or more” or “at least one” and indefinite articles such as “a” or“an” (e.g., “a” and/or “an” should typically be interpreted to mean “atleast one” or “one or more”); the same holds true for the use ofdefinite articles used to introduce claim recitations.

In addition, even if a specific number of an introduced claim recitationis explicitly recited, those skilled in the art will recognize that suchrecitation should typically be interpreted to mean at least the recitednumber (e.g., the bare recitation of “two recitations,” without othermodifiers, typically means at least two recitations, or two or morerecitations). Furthermore, in those instances where a conventionanalogous to “at least one of A, B, and C, etc.” is used, in generalsuch a construction is intended in the sense one having skill in the artwould understand the convention (e.g., “a system having at least one ofA, B, and C” would include but not be limited to systems that have Aalone, B alone, C alone, A and B together, A and C together, B and Ctogether, and/or A, B, and C together, etc.). In those instances where aconvention analogous to “at least one of A, B, or C, etc.” is used, ingeneral such a construction is intended in the sense one having skill inthe art would understand the convention (e.g., “a system having at leastone of A, B, or C” would include but not be limited to systems that haveA alone, B alone, C alone, A and B together, A and C together, B and Ctogether, and/or A, B, and C together, etc.). It will be furtherunderstood by those within the art that typically a disjunctive wordand/or phrase presenting two or more alternative terms, whether in thedescription, claims, or drawings, should be understood to contemplatethe possibilities of including one of the terms, either of the terms, orboth terms unless context dictates otherwise. For example, the phrase “Aor B” will be typically understood to include the possibilities of “A”or “B” or “A and B.”

With respect to the appended claims, those skilled in the art willappreciate that recited operations therein may generally be performed inany order. Also, although various operational flow diagrams arepresented in a sequence(s), it should be understood that the variousoperations may be performed in other orders than those which areillustrated, or may be performed concurrently. Examples of suchalternate orderings may include overlapping, interleaved, interrupted,reordered, incremental, preparatory, supplemental, simultaneous,reverse, or other variant orderings, unless context dictates otherwise.Furthermore, terms like “responsive to,” “related to,” or otherpast-tense adjectives are generally not intended to exclude suchvariants, unless context dictates otherwise.

It is worthy to note that any reference to “one aspect,” “an aspect,”“an exemplification,” “one exemplification,” and the like means that aparticular feature, structure, or characteristic described in connectionwith the aspect is included in at least one aspect. Thus, appearances ofthe phrases “in one aspect,” “in an aspect,” “in an exemplification,”and “in one exemplification” in various places throughout thespecification are not necessarily all referring to the same aspect.Furthermore, the particular features, structures or characteristics maybe combined in any suitable manner in one or more aspects.

Any patent application, patent, non-patent publication, or otherdisclosure material referred to in this specification and/or listed inany Application Data Sheet is incorporated by reference herein, to theextent that the incorporated materials is not inconsistent herewith. Assuch, and to the extent necessary, the disclosure as explicitly setforth herein supersedes any conflicting material incorporated herein byreference. Any material, or portion thereof, that is said to beincorporated by reference herein, but which conflicts with existingdefinitions, statements, or other disclosure material set forth hereinwill only be incorporated to the extent that no conflict arises betweenthat incorporated material and the existing disclosure material.

In summary, numerous benefits have been described which result fromemploying the concepts described herein. The foregoing description ofthe one or more forms has been presented for purposes of illustrationand description. It is not intended to be exhaustive or limiting to theprecise form disclosed. Modifications or variations are possible inlight of the above teachings. The one or more forms were chosen anddescribed in order to illustrate principles and practical application tothereby enable one of ordinary skill in the art to utilize the variousforms and with various modifications as are suited to the particular usecontemplated. It is intended that the claims submitted herewith definethe overall scope.

What is claimed is:
 1. An augmented reality display system for useduring a surgical procedure, the augmented reality display systemcomprising: an imaging device to capture a real image of a surgical areaduring the surgical procedure; an augmented reality display to presentan overlay of an operational aspect of a surgical instrument beingactively visualized onto the real image of the surgical area, whereinthe overlay combines aspects of tissue interaction in the surgical areawith functional data received from the surgical instrument; and aprocessor to: receive the functional data from the surgical instrument;determine the overlay related to the operational aspect of the surgicalinstrument; and combine the aspect of the tissue in the surgical areawith the functional data received from the surgical instrument.
 2. Theaugmented reality display system of claim 1, wherein the surgicalinstrument comprises an end effector with jaws to capture tissuetherebetween, wherein the operational aspect of the surgical instrumentis clamping tissue between the jaws of the end effector.
 3. Theaugmented reality display system of claim 2, wherein the aspect of thetissue is appropriate capture of the tissue between the jaws of the endeffector.
 4. The augmented reality display system of claim 2, whereinthe aspect of the tissue is incorrectly positioned tissue between thejaws of the end effector.
 5. The augmented reality display system ofclaim 2, wherein the aspect of the tissue is insufficiently capturedtissue between the jaws of the end effector.
 6. The augmented realitydisplay system of claim 2, wherein the aspect of the tissue is tensionof the tissue between the jaws of the end effector.
 7. The augmentedreality display system of claim 1, wherein the surgical instrumentcomprises an end effector with jaws to capture tissue therebetween andwherein the operational aspect of the surgical instrument isoverheating.
 8. The augmented reality display system of claim 1, whereinthe surgical instrument comprises an end effector with jaws to capturetissue therebetween and wherein the operational aspect of the surgicalinstrument is jaw closure position.
 9. The augmented reality displaysystem of claim 8, wherein the aspect of the tissue is movement or flowof the tissue between the jaws.
 10. The augmented reality display systemof claim 1, wherein the surgical instrument comprises an end effectorwith jaws to capture tissue therebetween and wherein the operationalaspect of the surgical instrument is clamping on metal or foreign objectbetween the jaws.
 11. The augmented reality display system of claim 10,wherein the aspect of the tissue is geometric relationship oftransections to the tissue and to other firings.
 12. The augmentedreality display system of claim 10, wherein the operational aspect ofthe surgical instrument is anvil orientation.
 13. The augmented realitydisplay system of claim 10, wherein the aspect of the tissue is tissuethickness.
 14. The augmented reality display system of claim 10, whereinthe operational aspect of the surgical instrument is clamping status ofthe jaws.
 15. The augmented reality display system of claim 1, whereinthe surgical instrument comprises an end effector with jaws to capturetissue therebetween and wherein the operational aspect of the surgicalinstrument is magnitude of the clamping status of the jaws.
 16. Theaugmented reality display system of claim 15, wherein the aspect of thetissue is tissue compression.
 17. The augmented reality display systemof claim 1, wherein the surgical instrument comprises an end effectorwith jaws to capture tissue therebetween and wherein the operationalaspect of the surgical instrument is sufficiency of clamping of thejaws.
 18. The augmented reality display system of claim 1, wherein thesurgical instrument comprises an end effector with jaws to capturetissue therebetween and wherein the operational aspect of the surgicalinstrument is firing sufficiency.
 19. The augmented reality displaysystem of claim 1, wherein the surgical instrument is an energy deviceand a tissue parameter is any one of impedances, cautery status,bleeding or magnitude, and wherein a functional aspect of the energydevice is any one of energy level, timing, or clamp pressure.
 20. Anaugmented reality display system for use during a surgical procedure,the augmented reality display system comprising: an imaging device tocapture a real image of a surgical area during the surgical procedure;an augmented reality display to present an overlay of an operationalaspect of a surgical instrument being actively visualized onto the realimage of the surgical area, wherein the overlay combines aspects oftissue interaction in the surgical area with functional data from thesurgical instrument; and a processor to: receive the functional datafrom the surgical instrument; determine the overlay related to theoperational aspect of the surgical instrument; combine the aspect of thetissue in the surgical area with the functional data received from thesurgical instrument; and configure the overlay.
 21. An augmented realitydisplay system for use during a surgical procedure, the augmentedreality display system comprising: an imaging device to capture a realimage of a surgical area during the surgical procedure; an augmentedreality display to present an overlay of an operational aspect of asurgical instrument being actively visualized onto the real image of thesurgical area, wherein the overlay combines aspects of tissueinteraction in the surgical area with functional data from the surgicalinstrument; and a processor to: receive the functional data from thesurgical instrument; determine the overlay related to the operationalaspect of the surgical instrument; combine the aspect of the tissue inthe surgical area with the functional data received from the surgicalinstrument; and display dynamic status of the surgical instrument.